[Feature Request] Better python dependency management

[byronz]

Is your feature request related to a problem? Please describe. this is mainly happening in the dash app repositories like dash-docs or dash gallery monorepo projects. Given a specific dash release, we don't really lock down the dependencies to a specific version. Taken the default requirements in require-install.txt as example

Flask>=1.0.2
flask-compress
plotly
dash_renderer==1.2.0
dash-core-components==1.5.0
dash-html-components==1.0.1
dash-table==4.5.0
future

we have a loose requirement on Flask, and we use the same txt file in dash-doc circleci setup as a cache key. we have renovate bot setup to some extent, but obviously it's not doing the job to keep dependencies updated in a more precise way. Our staging environment for dash-docs is deployed using heroku, this is where the problem starts to reveal that it also refers to the same txt file, but without cache, each time it will install a more recent version of flask until some regression bug happens in a new version, and we also have the same bug not detected in an earlier stage (CI), but in heroku.

@cldougl also have similar problem on on-prem side with the demo apps.

Describe the solution you'd like we need to at least lock/freeze the version during the dash release process ( @Marc-Andre-Rivet ) in a way that these versions can also be accessible and used by other apps to prevent any dependency conflict or bug happening.

the simplest way can update the requirements files with pip freeze during the release as part of the content of the release snapshot (both source code and release artifacts)

Describe alternatives you've considered

we can also share the system dependency management tool like poetry, it has a complete command toolchain to do the job from adding the dependency to publishing dash

[byronz]

@shammamah feel free to report your incident in dash-bio if you think it's relevant so we can have more incident reports from the dash eco-system.

[shammamah-zz]

The issue I was having with https://github.com/plotly/dash-bio/pull/434 is that the pip environment was loading from a cache that was made a couple weeks ago (October 23), so the version of dash was never updated. This led to a ContextualVersionConflict.

The 3.6 CI runs went smoothly, but the 3.7 run has an error that is to do with conflicting versions of dash (the cached version had 1.4.1, but the new version should have 1.6.0 as defined in the requirements.txt and the install_requires for the package).

I ran the CI commands locally on my machine (which is running 3.7) and they worked. I also SSHed into the CircleCI machine, removed the virtual environment, recreated it, and ran all the CI commands, and it worked. Removing the restore_cache step from my CircleCI config also stopped this issue from popping up.

I will attempt to resolve this by adding the Python version to the checksum as @byronz suggested.

[alexcjohnson]

The piece of this I'm personally unclear the right way to handle is that dash is a library folks will integrate with their own environments, where they may have other needs for Flask etc. We can lock down versions of the rest of the dash core because dash is the only way people should be accessing those, but I don't think we can say that about Flask since people using Dash within a larger Flask app may have their own requirements on that.

On the other hand, when we make a dash app we want all these deps locked down, and it would be nice to not need to know about and specifically list Dash internals like Flask.

What if we made another extras_require so in an app you can install something like dash[lock] and you get exact versions of all of these, that we keep up-to-date and tested?

[byronz]

playing with poetry to get a taste of its dep locks, here is the output of adding flask with poetry add flask

[tool.poetry.dependencies] flask = "^1.1"

[tool.poetry.dev-dependencies] pytest = "^3.0"

[[package]]
category = "dev"
description = "Atomic file writes."
name = "atomicwrites"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "1.3.0"

[[package]]
category = "dev"
description = "Classes Without Boilerplate"
name = "attrs"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "19.3.0"

[[package]]
category = "main"
description = "Composable command line interface toolkit"
name = "click"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "7.0"

[[package]]
category = "dev"
description = "Cross-platform colored terminal text."
marker = "sys_platform == \"win32\""
name = "colorama"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "0.4.1"

[[package]]
category = "main"
description = "A simple framework for building complex web applications."
name = "flask"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
version = "1.1.1"

[package.dependencies]
Jinja2 = ">=2.10.1"
Werkzeug = ">=0.15"
click = ">=5.1"
itsdangerous = ">=0.24"

[[package]]
category = "dev"
description = "Read metadata from Python packages"
marker = "python_version < \"3.8\""
name = "importlib-metadata"
optional = false
python-versions = ">=2.7,!=3.0,!=3.1,!=3.2,!=3.3"
version = "0.23"

[package.dependencies]
zipp = ">=0.5"

[[package]]
category = "main"
description = "Various helpers to pass data to untrusted environments and back."
name = "itsdangerous"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "1.1.0"

[[package]]
category = "main"
description = "A very fast and expressive template engine."
name = "jinja2"
optional = false
python-versions = "*"
version = "2.10.3"

[package.dependencies]
MarkupSafe = ">=0.23"

[[package]]
category = "main"
description = "Safely add untrusted strings to HTML/XML markup."
name = "markupsafe"
optional = false
python-versions = ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*"
version = "1.1.1"

[[package]]
category = "dev"
description = "More routines for operating on iterables, beyond itertools"
name = "more-itertools"
optional = false
python-versions = ">=3.4"
version = "7.2.0"

[[package]]
category = "dev"
description = "plugin and hook calling mechanisms for python"
name = "pluggy"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "0.13.0"

[package.dependencies]
[package.dependencies.importlib-metadata]
python = "<3.8"
version = ">=0.12"

[[package]]
category = "dev"
description = "library with cross-python path, ini-parsing, io, code, log facilities"
name = "py"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "1.8.0"

[[package]]
category = "dev"
description = "pytest: simple powerful testing with Python"
name = "pytest"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "3.10.1"

[package.dependencies]
atomicwrites = ">=1.0"
attrs = ">=17.4.0"
colorama = "*"
more-itertools = ">=4.0.0"
pluggy = ">=0.7"
py = ">=1.5.0"
setuptools = "*"
six = ">=1.10.0"

[[package]]
category = "dev"
description = "Python 2 and 3 compatibility utilities"
name = "six"
optional = false
python-versions = ">=2.6, !=3.0.*, !=3.1.*"
version = "1.13.0"

[[package]]
category = "main"
description = "The comprehensive WSGI web application library."
name = "werkzeug"
optional = false
python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*"
version = "0.16.0"

[[package]]
category = "dev"
description = "Backport of pathlib-compatible object wrapper for zip files"
marker = "python_version < \"3.8\""
name = "zipp"
optional = false
python-versions = ">=2.7"
version = "0.6.0"

[package.dependencies]
more-itertools = "*"

[metadata]
content-hash = "57f25d4b753e6a3ebaf53febf6a8e51339a8805c919670f145be34930b20c5ae"
python-versions = "^3.7"

[metadata.hashes]
atomicwrites = ["03472c30eb2c5d1ba9227e4c2ca66ab8287fbfbbda3888aa93dc2e28fc6811b4", "75a9445bac02d8d058d5e1fe689654ba5a6556a1dfd8ce6ec55a0ed79866cfa6"]
attrs = ["08a96c641c3a74e44eb59afb61a24f2cb9f4d7188748e76ba4bb5edfa3cb7d1c", "f7b7ce16570fe9965acd6d30101a28f62fb4a7f9e926b3bbc9b61f8b04247e72"]
click = ["2335065e6395b9e67ca716de5f7526736bfa6ceead690adf616d925bdc622b13", "5b94b49521f6456670fdb30cd82a4eca9412788a93fa6dd6df72c94d5a8ff2d7"]
colorama = ["05eed71e2e327246ad6b38c540c4a3117230b19679b875190486ddd2d721422d", "f8ac84de7840f5b9c4e3347b3c1eaa50f7e49c2b07596221daec5edaabbd7c48"]
flask = ["13f9f196f330c7c2c5d7a5cf91af894110ca0215ac051b5844701f2bfd934d52", "45eb5a6fd193d6cf7e0cf5d8a5b31f83d5faae0293695626f539a823e93b13f6"]
importlib-metadata = ["aa18d7378b00b40847790e7c27e11673d7fed219354109d0e7b9e5b25dc3ad26", "d5f18a79777f3aa179c145737780282e27b508fc8fd688cb17c7a813e8bd39af"]
itsdangerous = ["321b033d07f2a4136d3ec762eac9f16a10ccd60f53c0c91af90217ace7ba1f19", "b12271b2047cb23eeb98c8b5622e2e5c5e9abd9784a153e9d8ef9cb4dd09d749"]
jinja2 = ["74320bb91f31270f9551d46522e33af46a80c3d619f4a4bf42b3164d30b5911f", "9fe95f19286cfefaa917656583d020be14e7859c6b0252588391e47db34527de"]
markupsafe = ["00bc623926325b26bb9605ae9eae8a215691f33cae5df11ca5424f06f2d1f473", "09027a7803a62ca78792ad89403b1b7a73a01c8cb65909cd876f7fcebd79b161", "09c4b7f37d6c648cb13f9230d847adf22f8171b1ccc4d5682398e77f40309235", "1027c282dad077d0bae18be6794e6b6b8c91d58ed8a8d89a89d59693b9131db5", "24982cc2533820871eba85ba648cd53d8623687ff11cbb805be4ff7b4c971aff", "29872e92839765e546828bb7754a68c418d927cd064fd4708fab9fe9c8bb116b", "43a55c2930bbc139570ac2452adf3d70cdbb3cfe5912c71cdce1c2c6bbd9c5d1", "46c99d2de99945ec5cb54f23c8cd5689f6d7177305ebff350a58ce5f8de1669e", "500d4957e52ddc3351cabf489e79c91c17f6e0899158447047588650b5e69183", "535f6fc4d397c1563d08b88e485c3496cf5784e927af890fb3c3aac7f933ec66", "62fe6c95e3ec8a7fad637b7f3d372c15ec1caa01ab47926cfdf7a75b40e0eac1", "6dd73240d2af64df90aa7c4e7481e23825ea70af4b4922f8ede5b9e35f78a3b1", "717ba8fe3ae9cc0006d7c451f0bb265ee07739daf76355d06366154ee68d221e", "79855e1c5b8da654cf486b830bd42c06e8780cea587384cf6545b7d9ac013a0b", "7c1699dfe0cf8ff607dbdcc1e9b9af1755371f92a68f706051cc8c37d447c905", "88e5fcfb52ee7b911e8bb6d6aa2fd21fbecc674eadd44118a9cc3863f938e735", "8defac2f2ccd6805ebf65f5eeb132adcf2ab57aa11fdf4c0dd5169a004710e7d", "98c7086708b163d425c67c7a91bad6e466bb99d797aa64f965e9d25c12111a5e", "9add70b36c5666a2ed02b43b335fe19002ee5235efd4b8a89bfcf9005bebac0d", "9bf40443012702a1d2070043cb6291650a0841ece432556f784f004937f0f32c", "ade5e387d2ad0d7ebf59146cc00c8044acbd863725f887353a10df825fc8ae21", "b00c1de48212e4cc9603895652c5c410df699856a2853135b3967591e4beebc2", "b1282f8c00509d99fef04d8ba936b156d419be841854fe901d8ae224c59f0be5", "b2051432115498d3562c084a49bba65d97cf251f5a331c64a12ee7e04dacc51b", "ba59edeaa2fc6114428f1637ffff42da1e311e29382d81b339c1817d37ec93c6", "c8716a48d94b06bb3b2524c2b77e055fb313aeb4ea620c8dd03a105574ba704f", "cd5df75523866410809ca100dc9681e301e3c27567cf498077e8551b6d20e42f", "e249096428b3ae81b08327a63a485ad0878de3fb939049038579ac0ef61e17e7"]
more-itertools = ["409cd48d4db7052af495b09dec721011634af3753ae1ef92d2b32f73a745f832", "92b8c4b06dac4f0611c0729b2f2ede52b2e1bac1ab48f089c7ddc12e26bb60c4"]
pluggy = ["0db4b7601aae1d35b4a033282da476845aa19185c1e6964b25cf324b5e4ec3e6", "fa5fa1622fa6dd5c030e9cad086fa19ef6a0cf6d7a2d12318e10cb49d6d68f34"]
py = ["64f65755aee5b381cea27766a3a147c3f15b9b6b9ac88676de66ba2ae36793fa", "dc639b046a6e2cff5bbe40194ad65936d6ba360b52b3c3fe1d08a82dd50b5e53"]
pytest = ["3f193df1cfe1d1609d4c583838bea3d532b18d6160fd3f55c9447fdca30848ec", "e246cf173c01169b9617fc07264b7b1316e78d7a650055235d6d897bc80d9660"]
six = ["1f1b7d42e254082a9db6279deae68afb421ceba6158efa6131de7b3003ee93fd", "30f610279e8b2578cab6db20741130331735c781b56053c59c4076da27f06b66"]
werkzeug = ["7280924747b5733b246fe23972186c6b348f9ae29724135a6dfc1e53cea433e7", "e5f4a1f98b52b18a93da705a7458e55afb26f32bff83ff5d19189f92462d65c4"]
zipp = ["3718b1cbcd963c7d4c5511a8240812904164b7f381b647143a89d3b98f9bcd8e", "f06903e9f1f43b12d371004b4ac7b06ab39a44adc747266928ae6debfa7b3335"]

[byronz]

some examples of resolve problem error

[SolverProblemError]
The current project must support the following Python versions: ~2.7 || ^3.4
Because no versions of pytest match >5.0.0,<5.0.1 || >5.0.1,<5.1.0 || >5.1.0,<5.1.1 || >5.1.1,<5.1.2 || >5.1.2,<5.1.3 || >5.1.3,<5.2.0 || >5.2.0,<5.2.1 || >5.2.1,<5.2.2 || >5.2.2,<6.0.0
 and pytest (5.0.0) requires Python >=3.5, pytest is forbidden.
And because pytest (5.0.1) requires Python >=3.5, pytest is forbidden.
And because pytest (5.1.0) requires Python >=3.5
 and pytest (5.1.1) requires Python >=3.5, pytest is forbidden.
And because pytest (5.1.2) requires Python >=3.5
 and pytest (5.1.3) requires Python >=3.5, pytest is forbidden.
And because pytest (5.2.0) requires Python >=3.5
 and pytest (5.2.1) requires Python >=3.5, pytest is forbidden.
So, because pytest (5.2.2) requires Python >=3.5
 and dash depends on pytest (^5.0.0), version solving failed.

[SolverProblemError]
The current project must support the following Python versions: ~2.7 || ^3.3
Because no versions of flake8 match >3.7,<3.7.1 || >3.7.1,<3.7.2 || >3.7.2,<3.7.3 || >3.7.3,<3.7.4 || >3.7.4,<3.7.5 || >3.7.5,<3.7.6 || >3.7.6,<3.7.7 || >3.7.7,<3.7.8 || >3.7.8,<3.7.9 || >3.7.9,<4.0
 and flake8 (3.7.0) requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, flake8 is forbidden.
And because flake8 (3.7.1) requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*
 and flake8 (3.7.2) requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, flake8 is forbidden.
And because flake8 (3.7.3) requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*
 and flake8 (3.7.4) requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, flake8 is forbidden.
And because flake8 (3.7.5) requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*
 and flake8 (3.7.6) requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, flake8 is forbidden.
And because flake8 (3.7.7) requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*
 and flake8 (3.7.8) requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, flake8 is forbidden.
So, because flake8 (3.7.9) requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*
 and dash depends on flake8 (^3.7), version solving failed.

 [SolverProblemError]
The current project must support the following Python versions: ~2.7 || ^3.4
Because no versions of flask match >1.1,<1.1.1 || >1.1.1,<2.0
 and flask (1.1.0) requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, flask is forbidden.
So, because flask (1.1.1) requires Python >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*
 and dash depends on flask (^1.1), version solving failed.

and the more precise way to specify versions

[tool.poetry.dependencies]
python = "~2.7 || ^3.4"
future = "^0.18.2"
dash_renderer = "=1.2.0"
plotly = "^4.2"
pytest = [
    { version = "^5.0.0", optional = true, python = "^3.5" },
    { version = "^4.0.0", optional = true, python = "<3.5" }
]
selenium = { version = "^3.141.0", optional = true }
[tool.poetry.dev-dependencies]
flake8 = "^3.7"
mock = "^3.0"
coloredlogs = "^10.0"
fire = "^0.2.1"
dash_flow_example = "=0.0.5"
dash-dangerously-set-inner-html = "^0.0.2"
[tool.poetry.extras]
testing = ["pytest", "selenium"]

[merwok]

Another solution is combining pip with pip-tools. You define open-ended requirements in requirements.in (or any other name, can also have multiple files for runtime / devel) and run pip-compile to get a full list of all requirements with pinned, compatible versions in requirements.txt. Example use in a dash app: https://github.com/jeremymoreau/covid19mtl/pull/14/commits/8a88d04d206037ade35edcf5e876306d73a4f52c

The «compiled» requirements file works with vanilla pip, heroku, etc. Tools like dependabot know how to update requirements.in and/or requirements.txt. Depending on the project, it can be simpler to use pip-tools rather than switch wholesale to flit or poetry.

Note also that pip recently gained a real dependency solver that will avoid many bad or stopped installs.

[gvwilson]

Hi - we are tidying up stale issues and PRs in Plotly's public repositories so that we can focus on things that are most important to our community. If this issue is still a concern, please add a comment letting us know what recent version of our software you've checked it with so that I can reopen it and add it to our backlog. (Please note that we will give priority to reports that include a short reproducible example.) If you'd like to submit a PR, we'd be happy to prioritize a review, and if it's a request for tech support, please post in our community forum. Thank you - @gvwilson