PyPI does not allow direct references in package deps (this is new)

oberstet commented 1 year ago

the usual python dependencies hassles .. needs some adjusts. in a fresh cpy 3.9 venv:

pip install -U -r requirements-dev.txt =>

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

sphinx-rtd-theme 1.1.1 requires docutils<0.18, but you'll have docutils 0.19 which is incompatible.
sphinx-rtd-theme 1.1.1 requires sphinx<6,>=1.6, but you'll have sphinx 6.1.3 which is incompatible.
pytest-cov 4.0.0 requires pytest>=4.6, but you'll have pytest 3.10.1 which is incompatible.
docker-compose 1.29.2 requires PyYAML<6,>=3.10, but you'll have pyyaml 6.0 which is incompatible.
awscli 1.27.50 requires colorama<0.4.5,>=0.2.5, but you'll have colorama 0.4.6 which is incompatible.
awscli 1.27.50 requires docutils<0.17,>=0.10, but you'll have docutils 0.19 which is incompatible.
awscli 1.27.50 requires PyYAML<5.5,>=3.10, but you'll have pyyaml 6.0 which is incompatible.

pip install -e . =>

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
sphinx 6.1.3 requires Jinja2>=3.0, but you have jinja2 2.11.3 which is incompatible.
sphinx-rtd-theme 1.1.1 requires docutils<0.18, but you have docutils 0.19 which is incompatible.
sphinx-rtd-theme 1.1.1 requires sphinx<6,>=1.6, but you have sphinx 6.1.3 which is incompatible.
nbconvert 7.2.7 requires jinja2>=3.0, but you have jinja2 2.11.3 which is incompatible.
nbconvert 7.2.7 requires markupsafe>=2.0, but you have markupsafe 1.1.1 which is incompatible.
docker-compose 1.29.2 requires PyYAML<6,>=3.10, but you have pyyaml 6.0 which is incompatible.

tox has deprecated (without fallback) whitelist_externals for allowlist_externals - the world has waited for that!
tox doesn't allow / require to specifiy "." anymore:

tox -e  sphinx,flake8,mypy,yapf .
usage: tox [-h] [--colored {yes,no}] [-v | -q] [--exit-and-dump-after seconds] [-c file] [--workdir dir] [--root dir] [--runner {virtualenv}] [--version] [--no-provision [REQ_JSON]] [--no-recreate-provision] [-r] [-x OVERRIDE]
           {run,r,run-parallel,p,depends,de,list,l,devenv,d,config,c,quickstart,q,exec,e,legacy,le} ...
tox: error: unrecognized arguments: .
hint: if you tried to pass arguments to a command use -- to separate them from tox ones
make: *** [Makefile:90: test_quick] Fehler 2

oberstet commented 1 year ago

fantastic .. partially fixed in https://github.com/crossbario/crossbar/pull/2060:

(cp39_13) (base) oberstet@intel-nuci7:~/scm/crossbario/crossbar$ make test_quick 
tox -e  sphinx,flake8,mypy,yapf
sphinx: commands[0]> sphinx-build -nWT -b dummy ./docs ./docs/_build
Sphinx v6.1.3 in Verwendung

Initializing Spelling Checker 7.7.0
loading pickled environment... erledigt
Initiated sphinxcontrib-images backend: `sphinxcontrib_images_lightbox2.lightbox2:LightBox2`
building [mo]: targets for 0 po files that are out of date
writing output... 
building [dummy]: targets for 141 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
reading sources... 
Downloading remote images...
Copying static files for sphinxcontrib-images...[100%] lightbox2-customize/jquery-noconflict.js                                         
looking for now-outdated files... none found
preparing documents... erledigt
writing output... [100%] web_archive                                                                                                    
build abgeschlossen.

The dummy builder generates no files.
sphinx: commands[1]> sphinx-build -b spelling -q -d ./docs/_build/doctrees ./docs ./docs/_build/spelling

WARNING: Found 1714 misspelled words
sphinx: commands[2]> sphinx-build -b dirhtml ./docs ./docs/_build
Sphinx v6.1.3 in Verwendung

Initializing Spelling Checker 7.7.0
loading pickled environment... erledigt
Initiated sphinxcontrib-images backend: `sphinxcontrib_images_lightbox2.lightbox2:LightBox2`
building [mo]: targets for 0 po files that are out of date
writing output... 
building [dirhtml]: targets for 0 source files that are out of date
updating environment: 0 added, 0 changed, 0 removed
reading sources... 
Downloading remote images...
Copying static files for sphinxcontrib-images...[100%] lightbox2-customize/jquery-noconflict.js                                         
looking for now-outdated files... none found
no targets are out of date.
build abgeschlossen.

The HTML pages are in docs/_build.
sphinx: OK ✔ in 11.25 seconds
flake8: commands[0]> flake8 --exclude crossbar/shell/reflection --ignore=E501,E402,E722,E741,W503,W504,E126,E251 crossbar
flake8: OK ✔ in 4.34 seconds
mypy: commands[0]> mypy --exclude '(test_.*\.py)|(syntaxerror\.py)' --ignore-missing-imports crossbar
Success: no issues found in 215 source files
mypy: OK ✔ in 0.57 seconds
yapf: commands[0]> yapf --version
yapf 0.29.0
yapf: commands[1]> yapf -rd --style=/home/oberstet/scm/crossbario/crossbar/yapf.ini '--exclude=crossbar/shell/reflection/*' '--exclude=crossbar/master/database/*' '--exclude=crossbar/edge/database/*' --exclude=crossbar/shell/repl.py --exclude=crossbar/worker/test/examples/syntaxerror.py crossbar
  sphinx: OK (11.25=setup[0.13]+cmd[2.12,7.22,1.77] seconds)
  flake8: OK (4.34=setup[0.01]+cmd[4.33] seconds)
  mypy: OK (0.57=setup[0.01]+cmd[0.55] seconds)
  yapf: OK (151.41=setup[0.01]+cmd[0.08,151.32] seconds)
  congratulations :) (168.00 seconds)

oberstet commented 1 year ago

I am tired of the brokeness of python deps mgmt - anyways:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

pytest-cov 4.0.0 requires pytest>=4.6, but you'll have pytest 3.10.1 which is incompatible.
docker-compose 1.29.2 requires PyYAML<6,>=3.10, but you'll have pyyaml 6.0 which is incompatible.

oberstet commented 1 year ago

alright, down to one confluent/conflicting deps:

parsimonious
eth-abi

 ERROR: Cannot install autobahn, crossbar==23.1.1.dev1 and eth-abi==2.2.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    crossbar 23.1.1.dev1 depends on parsimonious>=0.10.0
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
    eth-abi 2.2.0 depends on parsimonious<0.9.0 and >=0.8.0
    crossbar 23.1.1.dev1 depends on parsimonious>=0.10.0
    eth-abi 3.0.1 depends on parsimonious<0.9.0 and >=0.8.0
    crossbar 23.1.1.dev1 depends on parsimonious>=0.10.0
    eth-abi 3.0.0 depends on parsimonious<0.9.0 and >=0.8.0
    crossbar 23.1.1.dev1 depends on parsimonious>=0.10.0
    eth-abi 2.1.1 depends on parsimonious<0.9.0 and >=0.8.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

Error: Process completed with exit code 1.

running locally

INFO: pip is looking at multiple versions of crossbar to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install autobahn, crossbar==23.1.1.dev1 and eth-abi==2.2.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    crossbar 23.1.1.dev1 depends on parsimonious>=0.10.0
    eth-abi 2.2.0 depends on parsimonious<0.9.0 and >=0.8.0
    crossbar 23.1.1.dev1 depends on parsimonious>=0.10.0
    eth-abi 3.0.1 depends on parsimonious<0.9.0 and >=0.8.0
    crossbar 23.1.1.dev1 depends on parsimonious>=0.10.0
    eth-abi 3.0.0 depends on parsimonious<0.9.0 and >=0.8.0
    crossbar 23.1.1.dev1 depends on parsimonious>=0.10.0
    eth-abi 2.1.1 depends on parsimonious<0.9.0 and >=0.8.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
py311-unpinned-trial: exit 1 (33.04 seconds) /home/oberstet/scm/crossbario/crossbar> pip install . pid=75427
py311-unpinned-trial: FAIL ✖ in 45.16 seconds

oberstet commented 1 year ago

above is all fixed, both locally and on CI, but unfortunately, when trying to publish to PyPI:

Uploading distributions to https://upload.pypi.org/legacy/
Uploading crossbar-23.1.1-py2.py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 873.5/873.5 kB • 00:01 • 1.6 MB/s
WARNING  Error during upload. Retry with the --verbose option for more details.                                                          
ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/                                                                 
         Invalid value for requires_dist. Error: Can't have direct dependency:                                                           
         'autobahn[compress,encryption,scram,serialization,twisted,xbr] @                                                                
         git+https://github.com/crossbario/autobahn-python.git@master#egg=autobahn'                                                      
make: *** [Makefile:51: publish] Fehler 1
(cpy311_5) oberstet@intel-nuci7:~/scm/crossbario/crossbar$

https://github.com/pypa/twine/issues/726 https://peps.python.org/pep-0440/#direct-references

crossbario / crossbar

PyPI does not allow direct references in package deps (this is new) #2061