Closed lucasfcnunes closed 12 months ago
arikfr
commented
1 year ago This is interesting, but is there really no cons?
One issue I can think of is that currently there are some conflicts between the many data source dependencies we use. The version of pip we're using manages to ignore/overcome this, but not sure if Poetry will?
Will you be interested in giving this a try (converting the requirements*.txt files into pyproject.toml) and seeing what the issues might be?
lucasfcnunes
commented
1 year ago Great take @arikfr!
Last week, as a draft, I did try to refactor with poetry and It went very well. It was too one of my concerns and I did not put it in the cons on purpose haha.
Let's head over to my conclusions after testing for packages conflicts.
The following pyproject.toml is conflict free and very close to the cat of our current req*.txt. Try it out with poetry update --lock.
# pyproject.toml
[tool.poetry]
name = "redash"
version = "10.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
license = "BSD 2"
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.8"
advocate = "^1.0.0"
aniso8601 = "^8.0.0"
# Install the dependencies of the bin/bundle-extensions script here.
# It has its own requirements file to simplify the frontend client build process
# Uncomment the requirement for ldap3 if using ldap.
# It is not included by default because of the GPL license conflict.
# ldap3 = "^2.2.4"
Authlib = "^0.15.5"
blinker = "^1.4"
click = "^6.7"
cryptography = "^2.8"
disposable-email-domains = "^0.0.52"
Flask = "^1.1.1"
Flask-Limiter = "^0.9.3"
Flask-Login = "^0.4.1"
flask-mail = "^0.9.1"
Flask-Migrate = "^2.5.2"
Flask-RESTful = "^0.3.7"
Flask-SQLAlchemy = "^2.4.1"
flask-talisman = "^0.7.0"
Flask-WTF = "^0.14.3"
funcy = "^1.13"
gevent = "^1.4.0"
greenlet = "^0.4.16"
gunicorn = "^20.0.4"
httplib2 = "^0.14.0"
itsdangerous = "^1.1.0"
Jinja2 = "^2.10.3"
jsonschema = "^3.1.1"
MarkupSafe = "^1.1.1"
maxminddb-geolite2 = "^2018.703"
parsedatetime = "^2.4"
passlib = "^1.7.1"
psycopg2 = "^2.8.3"
pycrypto = "^2.6.1"
PyJWT = "^1.7.1"
pyOpenSSL = "^19.0.0"
# We need to pin the version of pyparsing, as newer versions break SQLAlchemy-Searchable-10.0.6 (newer versions no longer depend on it)
pyparsing = "^2.1"
pypd = "^1.1.0"
pysaml2 = "^6.1.0"
pystache = "^0.5.4"
python-dateutil = "^2.8"
python-dotenv = "^1.0"
pytz = "^2022.7.1"
PyYAML = "^5.1.2"
redis = "^3.5.0"
requests = "^2.21.0"
RestrictedPython = "^5.0"
rq = "^1.13"
rq-scheduler = "^0.13"
semver = "^2.8.1"
sentry-sdk = "^0.14.3"
simplejson = "^3.16.0"
SQLAlchemy = "^1.3.10"
# We can't upgrade SQLAlchemy-Searchable version as newer versions require PostgreSQL > 9.6, but we target older versions at the moment.
SQLAlchemy-Searchable = "^0.10.6"
SQLAlchemy-Utils = "^0.34.2"
sqlparse = "^0.3.0"
sshtunnel = "^0.1.5"
statsd = "^3.3.0"
supervisor = "^4.1.0"
supervisor_checks = "^0.8.1"
ua-parser = "^0.8.0"
user-agents = "^2.0"
werkzeug = "^0.16.1"
wtforms = "^2.2.1"
xlsxwriter = "^1.2.2"
[tool.poetry.group.all_ds.dependencies]
atsd_client = "^3.0.5"
azure-kusto-data = "^0.0.35"
boto3 = "^1.10"
botocore = "^1.29"
cassandra-driver = "^3.21.0"
# certifi is needed to support MongoDB and SSL:
certifi = "^2019.9.11"
cmem-cmempy = "^21.2.3"
dql = "^0.6"
dynamo3 = "^1.0"
firebolt-sdk = "*"
google-api-python-client = "^1.7.11"
gspread = "^3.1.0"
ibm-db = "^2.0.9"
impyla = "^0.16.0"
influxdb = "^5.2.3"
memsql = "^3.0.0"
msal = "^1.21.0"
mysqlclient = "^1.3.14"
nzalchemy = "*"
nzpy = "^1.15"
oauth2client = "^4.1.3"
openpyxl = "^3.0.7"
pandas = "^1.5"
phoenixdb = "^0.7"
pinotdb = "^0.4.5"
protobuf = "^3.17.3"
pyarrow = "^11.0.0"
PyAthena = "^1.5.0"
pydgraph = "^2.0.2"
pydruid = "^0.5.7"
pyexasol = "^0.12.0"
pyhive = "^0.6.1"
# "Existing scripts should be migrated to pyomnisci from pymapd, this library will not be updated moving forward." - https://github.com/heavyai/pymapd
# pymapd = { version = "^0.26", python = ">=3.7,<3.9" }
pymongo = { version = "^3.9.0", extras = ["srv", "tls"] }
pymssql = "^2.2.7"
pyodbc = "^4.0.28"
python-arango = "^6.1.0"
python-rapidjson = "^0.8.0"
qds-sdk = "^1.9.6"
requests_aws_sign = "^0.1.5"
sasl = "^0.1.3"
simple_salesforce = "^0.74.3"
snowflake-connector-python = "^2.1.3"
td-client = "^1.0.0"
thrift = "^0.8.0"
thrift_sasl = "^0.1.0"
trino = "^0.305"
vertica-python = "^0.9.5"
xlrd = "^2.0.1"
[tool.poetry.group.test.dependencies]
coverage = "^4.5"
freezegun = "^0.3"
mock = "^3.0"
# PyMongo and Athena dependencies are needed for some of the unit tests:
# (this is not perfect and we should resolve this in a different way)
PyAthena = "^1.5"
pymongo = { version = "^3.9", extras = ["srv", "tls"] }
pytest = "^5.2"
pytest-cov = "^2.8"
[tool.poetry.group.dev.dependencies]
ptpython = "^3.0"
ptvsd = "^4.3"
watchdog = "^2.3"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"I'm closing as not planned. @arikfr
darkone23
commented
1 year ago +1
Poetry adds dependency pinning - without it every user is potentially downloading different versions of the projects dependencies.
Also agree old and insecure versions of python and packages should be left out of future releases. What are the defined and supported release targets?
This project is currently effectively using docker to pin release dependencies - but using poetry makes the python dependencies explicit.
justinclift
commented
1 year ago @guidopetri mentioned Poetry a few days ago in our Discord channel too.
I'm not personally familiar with it, so it's probably better for our Pythonistas to weigh in. (CC @getredash/maintainers)
Thoughts?
guidopetri
commented
1 year ago I'm a huge +1 of course :)
eradman
commented
12 months ago The switch to poetry was added in commit c97afeb327d8d54e7219ac439cc93d0f234763e5
Further refinements are addressed in https://github.com/getredash/redash/pull/6440
The python-poetry project has ~24k stars and is widely used in modern python projects.
Pros:
Cons: