Closed chongkong closed 3 years ago
I used third party dependency resolver (poetry) to resolve all abstract dependencies I have, and found a conflict. Here's the list:
absl-py >=0.9, <0.11
apache-airflow[mysql] >=1.10.10, <2
apache-beam[gcp] >=2.23, <3
attrs >=19.3.0, <20
click >=7, <8
docker >=4.1, <5
google-api-python-client >=1.7.11, <2
google-resumable-media >=1, <2
grpcio >=1.28.1, <2
ipython >=7, <8
ipywidgets >=7, <8
jinja2 >=2.7.3, <3
joblib >=0.12, <0.15
jupyter >=1, <2
keras-tuner >=1, <2
kfp >=0.4, <0.5
kubernetes >=10.0.1, <12
numpy >=1.16, <2
pandas >=1.0, <2
prompt-toolkit >=2.0.10, <3
protobuf >=3.9.2, <4
pyarrow >=0.17, <0.18
pydot >=1.2, <2
pytest >=5, <6
pyyaml >=3.12, <6
scikit-learn >=0.18, <0.24
scipy >=1.4.1, <2
six >=1.12, <2
tensorflow >=1.15.2, !=2.0.*, !=2.1.*, !=2.2.*, <2.4
tensorflow-serving-api >=1.15, !=2.0.*, !=2.1.*, !=2.2.*, <3
tensorflowjs >=2.0.1.post1, <3
WTForms ==2.2.1
The conflict occurs on prompt-toolkit
where:
we're installing tfx[tfjs]
, tensorflow-model-analysis
(from master HEAD)
tensorflow-model-analysis requires prompt-toolkit>=2.0.10,<3
It took only 36.9 seconds to find the conflict. I would rephrase this issue as "very slow (infinite loop?) resolution when having a conflict, with not helpful logging"
I re-wordrd the title to try reflecting the issue at hand here.
I’ve been trying to work on this, but found it difficult to come up with a test case to compare various implementations. Would it be able for you to provide some sort of workflow or requirements.txt that I can run pip install --use-feature=2020-resolver
repeatedly against?
The one I mentioned in the seconds comments is a one sample (as a requirements.txt) that stucks when running pip resolver. (checked now). The message is not exactly the same as the first one though
% pip install --use-feature=2020-resolver -r requirements.txt
Collecting attrs<20,>=19.3.0
Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting google-resumable-media<2,>=1
Using cached google_resumable_media-1.0.0-py2.py3-none-any.whl (42 kB)
Collecting jupyter<2,>=1
Using cached jupyter-1.0.0-py2.py3-none-any.whl (2.7 kB)
Processing /Users/jjong/Library/Caches/pip/wheels/3a/53/69/8b0da5e84103f46d355064f87a3df7067c935bec3ad880d3a1/kfp-0.4.0-py3-none-any.whl
Collecting prompt-toolkit<3,>=2.0.10
Using cached prompt_toolkit-2.0.10-py3-none-any.whl (340 kB)
Collecting WTForms==2.2.1
Using cached WTForms-2.2.1-py2.py3-none-any.whl (166 kB)
Collecting google-crc32c<2.0dev,>=1.0
Using cached google_crc32c-1.0.0-cp37-cp37m-macosx_10_9_x86_64.whl (28 kB)
Processing /Users/jjong/Library/Caches/pip/wheels/a9/4b/fd/cdd013bd2ad1a7162ecfaf954e9f1bb605174a20e3c02016b7/argo_models-2.2.1a0-py3-none-any.whl
Collecting absl-py<0.11,>=0.9
Using cached absl_py-0.10.0-py3-none-any.whl (127 kB)
Collecting apache-beam[gcp]<3,>=2.23
Using cached apache_beam-2.24.0-cp37-cp37m-macosx_10_9_x86_64.whl (3.6 MB)
Processing /Users/jjong/Library/Caches/pip/wheels/dc/9a/e9/49e627353476cec8484343c4ab656f1e0d783ee77b9dde2d1f/crcmod-1.7-cp37-cp37m-macosx_10_15_x86_64.whl
Processing /Users/jjong/Library/Caches/pip/wheels/a4/61/fd/c57e374e580aa78a45ed78d5859b3a44436af17e22ca53284f/dill-0.3.1.1-py3-none-any.whl
Collecting requests<3.0.0,>=2.24.0
Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Processing /Users/jjong/Library/Caches/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0/future-0.18.2-py3-none-any.whl
Collecting google-cloud-language<2,>=1.3.0
Using cached google_cloud_language-1.3.0-py2.py3-none-any.whl (83 kB)
Collecting grpcio-gcp<1,>=0.2.2
Using cached grpcio_gcp-0.2.2-py2.py3-none-any.whl (9.4 kB)
Processing /Users/jjong/Library/Caches/pip/wheels/19/b5/2f/1cc3cf2b31e7a9cd1508731212526d9550271274d351c96f16/google_apitools-0.5.31-py3-none-any.whl
Processing /Users/jjong/Library/Caches/pip/wheels/0b/cf/2f/1a1749d3a3650fac3305a8d7f9237b6de7c41068e2f8520ca2/keras_tuner-1.0.1-py3-none-any.whl
Collecting pyarrow<0.18,>=0.17
Using cached pyarrow-0.17.1-cp37-cp37m-macosx_10_9_intel.whl (42.0 MB)
Collecting tensorflow-serving-api!=2.0.*,!=2.1.*,!=2.2.*,<3,>=1.15
Using cached tensorflow_serving_api-2.3.0-py2.py3-none-any.whl (38 kB)
Collecting tensorflow!=2.0.*,!=2.1.*,!=2.2.*,<2.4,>=1.15.2
Downloading tensorflow-2.3.1-cp37-cp37m-macosx_10_9_x86_64.whl (165.1 MB)
|████████████████████████████████| 165.1 MB 10.7 MB/s
Requirement already satisfied: wheel>=0.26 in ./venv/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,<2.4,>=1.15.2->-r requirements.txt (line 29)) (0.35.1)
Collecting astunparse==1.6.3
Using cached astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Requirement already satisfied: wheel>=0.26 in ./venv/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,<2.4,>=1.15.2->-r requirements.txt (line 29)) (0.35.1)
Collecting tensorflow-estimator<2.4.0,>=2.3.0
Using cached tensorflow_estimator-2.3.0-py2.py3-none-any.whl (459 kB)
Collecting h5py<2.11.0,>=2.10.0
Using cached h5py-2.10.0-cp37-cp37m-macosx_10_6_intel.whl (3.0 MB)
Collecting tensorboard<3,>=2.3.0
Using cached tensorboard-2.3.0-py3-none-any.whl (6.8 MB)
Requirement already satisfied: setuptools>=41.0.0 in ./venv/lib/python3.7/site-packages (from tensorboard<3,>=2.3.0->tensorflow!=2.0.*,!=2.1.*,!=2.2.*,<2.4,>=1.15.2->-r requirements.txt (line 29)) (50.3.0)
Requirement already satisfied: wheel>=0.26 in ./venv/lib/python3.7/site-packages (from tensorflow!=2.0.*,!=2.1.*,!=2.2.*,<2.4,>=1.15.2->-r requirements.txt (line 29)) (0.35.1)
Collecting gast==0.3.3
Using cached gast-0.3.3-py2.py3-none-any.whl (9.7 kB)
Processing /Users/jjong/Library/Caches/pip/wheels/7c/06/54/bc84598ba1daf8f970247f550b175aaaee85f68b4b0c5ab2c6/termcolor-1.1.0-cp37-none-any.whl
Collecting google-auth-oauthlib<0.5,>=0.4.1
Using cached google_auth_oauthlib-0.4.1-py2.py3-none-any.whl (18 kB)
Processing /Users/jjong/Library/Caches/pip/wheels/01/da/49/14b780ebf5253d1143357b65d234b44134d5aa255bc1f2f59f/kfp_server_api-0.3.0-py3-none-any.whl
Collecting python-dateutil<3,>=2.8.0
Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting cachetools<4,>=3.1.0
Using cached cachetools-3.1.1-py2.py3-none-any.whl (11 kB)
Collecting keras-preprocessing<1.2,>=1.1.1
Using cached Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
Collecting google-pasta>=0.1.8
Using cached google_pasta-0.2.0-py3-none-any.whl (57 kB)
Collecting apache-airflow[mysql]<2,>=1.10.10
Using cached apache_airflow-1.10.12-py2.py3-none-any.whl (4.7 MB)
Collecting flask-caching<1.4.0,>=1.3.3
Using cached Flask_Caching-1.3.3-py2.py3-none-any.whl (15 kB)
Processing /Users/jjong/Library/Caches/pip/wheels/1a/f4/8a/a5024fb77b32ed369e5c409081e5f00fbe3b92fdad653f6e69/unicodecsv-0.14.1-py3-none-any.whl
Processing /Users/jjong/Library/Caches/pip/wheels/42/0e/a4/5290cd4d309d756617f4d8eedd60813653d606e21ccaf7f286/python_nvd3-0.15.0-py3-none-any.whl
Collecting tenacity==4.12.0
Using cached tenacity-4.12.0-py2.py3-none-any.whl (25 kB)
Processing /Users/jjong/Library/Caches/pip/wheels/e2/0e/dc/22a77318fe07bfdde46e434ca3a8133fb5016f396700ad7021/json_merge_patch-0.2-py3-none-any.whl
Processing /Users/jjong/Library/Caches/pip/wheels/41/d4/fa/31df9cc56ee41572a4dfc1be7dfc87df59a5d6a7fd03e97eab/pendulum-1.4.4-cp37-cp37m-macosx_10_15_x86_64.whl
Processing /Users/jjong/Library/Caches/pip/wheels/f9/71/7e/5a0e31a65d8f244dec9c3febd3ef18a38d7fe4767fdf709011/Flask_Admin-1.5.4-py3-none-any.whl
Collecting cattrs~=1.0
Using cached cattrs-1.0.0-py2.py3-none-any.whl (14 kB)
Processing /Users/jjong/Library/Caches/pip/wheels/6a/76/b4/f97e26f8f60b5b2b1287415f1202e2b34a97775cead74072be/tzlocal-1.5.1-py3-none-any.whl
Collecting sqlalchemy-jsonfield~=0.9
Using cached SQLAlchemy_JSONField-0.9.0-py2.py3-none-any.whl (10 kB)
Collecting colorlog==4.0.2
Using cached colorlog-4.0.2-py2.py3-none-any.whl (17 kB)
Processing /Users/jjong/Library/Caches/pip/wheels/26/31/14/a8dcde240ba2a1e052d0d3a45e09c7d2843922474c8c163d5c/flask_swagger-0.2.14-py3-none-any.whl
Collecting iso8601>=0.1.12
Using cached iso8601-0.1.13-py3-none-any.whl (9.3 kB)
Collecting cached-property~=1.5
Downloading cached_property-1.5.2-py2.py3-none-any.whl (7.6 kB)
Collecting flask-wtf<0.15,>=0.14.2
Using cached Flask_WTF-0.14.3-py2.py3-none-any.whl (13 kB)
Collecting kubernetes<12,>=10.0.1
Using cached kubernetes-11.0.0-py3-none-any.whl (1.5 MB)
Requirement already satisfied: setuptools>=41.0.0 in ./venv/lib/python3.7/site-packages (from tensorboard<3,>=2.3.0->tensorflow!=2.0.*,!=2.1.*,!=2.2.*,<2.4,>=1.15.2->-r requirements.txt (line 29)) (50.3.0)
Collecting protobuf<4,>=3.9.2
Using cached protobuf-3.13.0-cp37-cp37m-macosx_10_9_x86_64.whl (1.3 MB)
Requirement already satisfied: setuptools>=41.0.0 in ./venv/lib/python3.7/site-packages (from tensorboard<3,>=2.3.0->tensorflow!=2.0.*,!=2.1.*,!=2.2.*,<2.4,>=1.15.2->-r requirements.txt (line 29)) (50.3.0)
Collecting requests-toolbelt>=0.8.0
Using cached requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)
Collecting chardet<4,>=3.0.2
Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting fasteners>=0.14
Using cached fasteners-0.15-py2.py3-none-any.whl (23 kB)
Collecting tensorboard-plugin-wit>=1.6.0
Using cached tensorboard_plugin_wit-1.7.0-py3-none-any.whl (779 kB)
Collecting flask<2.0,>=1.1.0
Using cached Flask-1.1.2-py2.py3-none-any.whl (94 kB)
Collecting itsdangerous
Using cached itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Processing /Users/jjong/Library/Caches/pip/wheels/c2/fc/58/bd368d9483b36eafa3673e2acd8dcd121b0695d10719fe5fce/setproctitle-1.1.10-cp37-cp37m-macosx_10_15_x86_64.whl
Collecting funcsigs<2.0.0,>=1.0.0
Using cached funcsigs-1.0.2-py2.py3-none-any.whl (17 kB)
Collecting click<8,>=7
Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting grpcio<2,>=1.28.1
Using cached grpcio-1.32.0-cp37-cp37m-macosx_10_9_x86_64.whl (3.3 MB)
Collecting scipy<2,>=1.4.1
Using cached scipy-1.5.2-cp37-cp37m-macosx_10_9_x86_64.whl (28.7 MB)
Collecting six<2,>=1.12
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting tensorflowjs<3,>=2.0.1.post1
Using cached tensorflowjs-2.4.0-py3-none-any.whl (61 kB)
Processing /Users/jjong/Library/Caches/pip/wheels/89/3b/7b/8b3cc8ac47137eabaeb6937a3ff0d33e78a12e2ba1e3ad4ba1/PyInquirer-1.0.3-py3-none-any.whl
Collecting tensorflowjs<3,>=2.0.1.post1
Using cached tensorflowjs-2.3.0-py3-none-any.whl (61 kB)
Collecting tensorflowjs<3,>=2.0.1.post1
Using cached tensorflowjs-2.1.0-py3-none-any.whl (60 kB)
Collecting tensorflowjs<3,>=2.0.1.post1
Using cached tensorflowjs-2.0.1.post1-py3-none-any.whl (60 kB)
(stuck forever..)
Can you share the requirements.txt’s content? I can’t craft one that exhibits the problem.
Hmm, so there is a bug in here. The resolver is trying to pin tensorflowjs 2.0.1.post1 and 2.1.0 repeated. Neither of them is satosfactory, but for some reason the resolver does not know it should give up. So bottom line is, the set of requirements here have internal conflicts and can never resolve, you’ll need to fix that. But the resolver isn’t correctly detecting the conflict, which is a bug. Not sure why that’s happening, could be a logical bug in the resolver or a misimplementation in the provider.
The next thing to do here is to come up with a minimal reproducible; the current requirements list is way too big and would take too long to debug.
This is the same as #9011, not #8713 -- the new resolver is expected to try different versions of a package during backtracking. What it's not expected to do, is be stuck between trying to pin the exact same package versions, as mentioned by @uranusjr above.
The resolver is trying to pin tensorflowjs 2.0.1.post1 and 2.1.0 repeated.
I'll close this as a duplicate of #9011, since it has more discussion related to the incorrect pinning behaviour.
What did you want to do?
Installing
tfx
package with head versions of some dependencies.Due to the Google's monolithic code system, during the development
tfx
package is depending on the HEAD version of some internal dependencies, such astensorflow-data-validation
,tensorflow-model-analysis
, etc. (There are total 5 depending libraries). We're installing head versions by building wheels from the source code of dependencies, and then pass all wheels to thepip install
command. So, this is a bit hard to reproduce, but I hope we can get some debugging idea or the possible failure cause from the deceptive error logs.Output
Additional information