search

pcdshub / pcds-envs

conda environments used by PCDS
Other
1 stars 9 forks source link

Python 3.10 waiting list/status #205

Open ZLLentz opened 2 years ago

ZLLentz commented 2 years ago

I'm going to periodically check the travis cron to keep up with which packages are preventing us from getting to python 3.10, updating here.

11/8/2021 status:

ZLLentz commented 2 years ago

Reminder to self: by the time 3.10 is ready, it'll also be time to drop 3.7 support (scheduled for end of 2021 in NEP 29) This will require a PR to pcds-ci-helpers such that:

ZLLentz commented 2 years ago

11/18/2021 status:

klauer commented 2 years ago

That's pretty great... maybe we should get back to pyca, then. It'd be really cool - but wholly unnecessary at this point - to have a pcds-3.10 env to poke around with.

ZLLentz commented 2 years ago

I've been considering if it's worth specifying alternate env builds for testing/poking around with that remove the packages with low compatibility. Some examples might be:

I'm not sure the cleanest way to do these

ZLLentz commented 2 years ago

6/6/2022 status is this perplexing message:

Encountered problems while solving:
  - package docutils-0.16-py27h8c360ce_1 requires python_abi 2.7.* *_cp27mu, but none of the providers can be installed

I think all previously known incompatibilities have been resolved

ZLLentz commented 2 years ago

After removing the docutils pin:

Encountered problems while solving:
  - package bluesky-queueserver-0.0.10-py37h89c1867_0 requires python_abi 3.7.* *_cp37m, but none of the providers can be installed
ZLLentz commented 2 years ago

I guess we still have dependencies that haven't made any py3.10 builds oh well

ZLLentz commented 2 years ago

py310 now fails on ipython which doesn't make any sense to me since it has a py310 build

ZLLentz commented 1 year ago

The build errors are still confusing. The latest try gives:

Encountered problems while solving:
  - package epicscorelibs-7.0.6.99.2.0-py37hb62d391_0 requires python_abi 3.7.* *_cp37m, but none of the providers can be installed

Like the previous failure, this is a package that has an available python 3.10 version. I suspect something that depends on epicscorelibs must have some strict pinning that excludes python 3.10.

ZLLentz commented 1 year ago

After quick investigation, this is not the case. All the packages that depend on epicscorelibs are installable in 3.10. I'm going to detour myself and create a script to figure out what's going on because I want to be allowed to do py3.10 at some point.

ZLLentz commented 1 year ago

The following packages on our conda packages list can't be installed individually along with py310:

For everything else, conda lets you do the install.

klauer commented 1 year ago

hxrsnd is noarch - what's happening there?

ZLLentz commented 1 year ago

Ok actually my script was really crude and hxrsnd passes on retry

It's just the precise tag on scikit-image that holds us to py39. That means we can remove the pin and deploy a beta py10 env to try out whenever we want to.

ZLLentz commented 1 year ago

Ok there is actually something to the hxrsnd one, you can install it on 3.10 but not the latest tag on 3.10 because the recipe explicitly forbids it :facepalm: https://github.com/pcdshub/hxrsnd/blob/5e1201bbb43d535d8c30a4fbf967c253b9d7dd00/conda-recipe/meta.yaml#L17

ZLLentz commented 1 year ago

It looks like a misguided past me decided that it was better to set the recipe upper pin than to engage with the issues. Mistake.

ZLLentz commented 1 year ago

Resulting rabbit hole leads to moved imports in python's collections module https://github.com/pcdshub/hxrsnd/pull/122#issuecomment-1309210810

ZLLentz commented 1 year ago

Is it now possible to create a python 3.10 environment! Let's run some tests on it and potentially deploy a py310 folder with pcds-6.0.0 for beta testing.

ZLLentz commented 1 year ago

First pass at the tests fails on some technicalities without making much headway: /cds/home/z/zlentz/github/pcds-envs/scripts/test_py310_001.txt But it's a start

klauer commented 1 year ago

From the logs, it looks like pytest isn't even running? At least, I don't see any pytest output and it exits with code 1, hmm

ZLLentz commented 1 year ago

Yeah it's a silent version conflict between pytest-qt and pytest. I thought this was only in an old version so I'm surprised to see it back. We probably have one or both of these pinned and it's causing an issue.

The other break at the bottom of the log is the pyca/epicscorelibs fiasco

ZLLentz commented 1 year ago

The latest pyca and p4p modules currently require different incompatible versions of epicscorelibs. First, mamba installs the one that pyca needs, then pip removes it in favor of the one p4p needs.

ZLLentz commented 1 year ago 
The following packages failed all retries:
tests/lucid
tests/pcdswidgets
tests/pyca
tests/typhos
The following packages had race conditions, but passed at least once:
tests/lightpath
Skipping extra-tests.sh, did not input an environment name as argument 1
pyca 3.2.0 has requirement epicscorelibs<7.0.6.99.3,>=7.0.6.99.2.0, but you have epicscorelibs 7.0.7.99.0.0.

py3.10: conda list | grep qt

pyqt                      5.12.3          py310hff52083_8    conda-forge
pyqt-impl                 5.12.3          py310h1f8e252_8    conda-forge
pyqt5-sip                 4.19.18         py310h122e73d_8    conda-forge
pyqtads                   3.8.2           py310hd8f1fbe_0    conda-forge
pyqtchart                 5.12            py310hfcd6d55_8    conda-forge
pyqtgraph                 0.12.4             pyhd8ed1ab_0    conda-forge
pyqtwebengine             5.12.1          py310hfcd6d55_8    conda-forge
pytest-qt                 4.2.0              pyhd8ed1ab_0    conda-forge
qt                        5.12.9               h1304e3e_6    conda-forge
qtawesome                 1.2.1              pyhd8ed1ab_0    conda-forge
qtconsole                 5.4.0              pyhd8ed1ab_0    conda-forge
qtconsole-base            5.4.0              pyha770c72_0    conda-forge
qtpy                      2.1.0              pyhd8ed1ab_0    conda-forge
qtpyinheritance           0.0.2              pyhd8ed1ab_0    conda-forge
sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge

py3.9: conda list | grep qt

conda list | grep qt
pyqt                      5.12.3           py39hf3d152e_8    conda-forge
pyqt-impl                 5.12.3           py39hde8b62d_8    conda-forge
pyqt5-sip                 4.19.18          py39he80948d_8    conda-forge
pyqtads                   3.8.2            py39h5a03fae_0    conda-forge
pyqtchart                 5.12             py39h0fcd23e_8    conda-forge
pyqtgraph                 0.12.4             pyhd8ed1ab_0    conda-forge
pyqtwebengine             5.12.1           py39h0fcd23e_8    conda-forge
pytest-qt                 3.3.0                      py_0    conda-forge
qt                        5.12.9               h1304e3e_6    conda-forge
qtawesome                 1.1.1              pyhd8ed1ab_0    conda-forge
qtconsole                 5.3.1              pyhd8ed1ab_0    conda-forge
qtconsole-base            5.3.1              pyha770c72_0    conda-forge
qtpy                      2.1.0              pyhd8ed1ab_0    conda-forge
qtpyinheritance           0.0.2              pyhd8ed1ab_0    conda-forge
sphinxcontrib-qthelp      1.0.3                      py_0    conda-forge
ZLLentz commented 1 year ago

Based on the above, it's probably safe to expand the standard test suites to check py3.10, but it can also wait until we resolve these items.

ZLLentz commented 1 year ago

There is a pattern for python 3.10-related failures in the qt modules: whenever the qt docs says that something must be an int, they now must be an int and not a float, or else bad things happen. Previously it looks like something was converting these to ints for us.

This covers all the test failures and the interactive failures I've observed so far, except for typhos's segfault which I haven't looked at yet.

Here's an example from clicking around in lucid:

[2022-11-17 09:28:20,394] [ERROR   ] - An uncaught exception happened: arguments did not match any overloaded call:
  QPoint(): too many arguments
  QPoint(int, int): argument 1 has unexpected type 'float'
  QPoint(QPoint): argument 1 has unexpected type 'float'
Traceback (most recent call last):
  File "/cds/home/z/zlentz/miniconda3/envs/pcds-6.0.0/bin/lucid", line 10, in <module>
    sys.exit(main())
  File "/cds/home/z/zlentz/github/lucid/lucid/launcher.py", line 255, in main
    launch(**kwargs)
  File "/cds/home/z/zlentz/github/lucid/lucid/launcher.py", line 243, in launch
    app.exec_()
  File "/cds/home/z/zlentz/miniconda3/envs/pcds-6.0.0/lib/python3.10/site-packages/pydm/widgets/byte.py", line 46, in paintEvent
    self._painter.drawEllipse(QPoint(w / 2.0, h / 2.0), r, r)
TypeError: arguments did not match any overloaded call:
  QPoint(): too many arguments
  QPoint(int, int): argument 1 has unexpected type 'float'
  QPoint(QPoint): argument 1 has unexpected type 'float'
[2022-11-17 09:28:20,560] [INFO    ] - Saved screenshot: /tmp/43a651da.1.png
[2022-11-17 09:28:22,740] [INFO    ] - Saved screenshot: /tmp/43a651da.2.LUCID_-_TMO.png
QPainter::begin: Painter already active
Segmentation fault (core dumped)

I don't know the best way to go about tracking these down or if there's some other way to address this. I'll put patches in for lucid and pcdswidgets since I identified it already but after that I want to slow down and consider systematic ways to either fix this everywhere or if there's some better way to approach the problem.

ZLLentz commented 1 year ago

We can't update to 3.10 until we either fix all the places this happens in pydm, typhos, etc. or find a different way to address it (different libraries? versions? builds?)

ZLLentz commented 1 year ago

I have not yet identified precisely which change is causing the new behavior.

klauer commented 1 year ago

Seems likely that it'd be in sip (the C++ wrapping tool which backs pyqt5). This will be a difficult one to fix generally and not in a whack-a-mole style - perhaps our best tool will be static analysis with pyright/mypy/etc.

ZLLentz commented 1 year ago

I agree that whack-a-mole isn't great here. I'm whacking a few obvious ones today under the principle that I can't unsee them but I'm not going to charge forward further without a tool.

ZLLentz commented 1 year ago

Linking the root cause we found in slack for completeness: https://docs.python.org/3/whatsnew/3.10.html#other-language-changes

Builtin and extension functions that take integer arguments no longer accept Decimals, Fractions and other objects that can be converted to integers only with a loss (e.g. that have the int() method but do not have the index() method). (Contributed by Serhiy Storchaka in bpo-37999.)

I've also now noticed https://docs.python.org/3/whatsnew/3.10.html#porting-to-python-3-10 which doesn't mention the above but is worth a quick skim

ZLLentz commented 1 year ago

The only blockage for this now (pending test suite failures) is the previous comment. A python 3.10 build now can be created with no environment-level build issues.