search

DeepLabCut / DeepLabCut

Official implementation of DeepLabCut: Markerless pose estimation of user-defined features with deep learning for all animals incl. humans
http://deeplabcut.org
GNU Lesser General Public License v3.0
4.65k stars 1.66k forks source link

Upper limits for package requirements #2746

Open tabedzki opened 1 month ago

tabedzki commented 1 month ago

Is there an existing issue for this?

Bug description

I am trying to install the latest version of deeplabcut==2.3.10 on Python 3.12 which has a limit on tensorflow>=2.0,<=2.10. This upper limit on tensorflow causes the dependency solver to go back to deeplabcut==2.2.3 even in a new virtual environment to comply with upper limit.

The ask

What is the reason for these upper limits packaging upper limits? Can they be safely removed or is there a functionality that deeplabcut depends on that broke after 2.10?

Related

Additionally, there are different package requirements between requirements.txt and setup.py. Can those be sync'ed or can be one the resource of truth and other one removed? Can those upper limits also be removed if they are not known to break anything?

requirements.txt

matplotlib<=3.5.2
tensorflow>=2.0,<2.13.0
tensorpack==0.11
tf_slim==1.1.0
torch==1.12

Setup.py

"matplotlib>=3.3,<3.9,!=3.7.0,!=3.7.1",
"tensorflow>=2.0,<=2.10",
"tensorpack>=0.11",
"torch",
"tf_slim>=1.1.0"

Operating System

Linux

DeepLabCut version

trying to install latest

Steps To Reproduce

Install python 3.12

  1. uv venv
  2. source .venv/bin/activate
  3. uv pip install deeplabcut[tf]==2.3.10

Relevant log output

× No solution found when resolving dependencies:
  ╰─▶ Because only the following versions of tensorflow are available:
          tensorflow<=2.5.3
          tensorflow>=2.6.0,<=2.6.5
          tensorflow>=2.7.0,<=2.7.4
          tensorflow>=2.8.0,<=2.8.4
          tensorflow>=2.9.0,<=2.9.3
          tensorflow>=2.10.0
      and tensorflow==2.0.0 has no wheels with a matching Python implementation tag, we can conclude that all of:
          tensorflow>=2.0.0,<2.0.1
          tensorflow>2.5.3,<2.6.0
          tensorflow>2.6.5,<2.7.0
          tensorflow>2.7.4,<2.8.0
          tensorflow>2.8.4,<2.9.0
          tensorflow>2.9.3,<2.10.0
       cannot be used.
      And because tensorflow>=2.0.1,<=2.0.4 has no wheels with a matching Python ABI tag, we can conclude that all of:
          tensorflow>=2.0.0,<2.0.2
          tensorflow>2.5.3,<2.6.0
          tensorflow>2.6.5,<2.7.0
          tensorflow>2.7.4,<2.8.0
          tensorflow>2.8.4,<2.9.0
          tensorflow>2.9.3,<2.10.0
       cannot be used.
      And because tensorflow==2.1.0 has no wheels with a matching Python implementation tag and all of:
          tensorflow>=2.1.1,<=2.5.3
          tensorflow>=2.6.0,<=2.6.5
          tensorflow>=2.7.0,<=2.7.4
          tensorflow>=2.8.0,<=2.8.4
          tensorflow>=2.9.0,<=2.9.3
          tensorflow==2.10.0
      have no wheels with a matching Python ABI tag, we can conclude that all of:
          tensorflow>=2.0.0,<2.1.2
          tensorflow>2.5.3,<2.6.0
          tensorflow>2.6.5,<2.7.0
          tensorflow>2.7.4,<2.8.0
          tensorflow>2.8.4,<2.9.0
          tensorflow>2.9.3,<2.10.0
       cannot be used.
      And because deeplabcut[tf]==2.3.10 depends on tensorflow>=2.0,<=2.10 and you require deeplabcut[tf]==2.3.10, we can conclude
      that your requirements are unsatisfiable.

      hint: Pre-releases are available for tensorflow in the requested range (e.g., 2.10.0rc3), but pre-releases weren't enabled (try:
      `--prerelease=allow`)

Code of Conduct

tabedzki commented 1 month ago

After making the original post I saw that there are things that changed in tensorflow that necessitated the limit and that there is work being done to make 3.0 a thing. However, the limit of packaging still affects 3.0.

The upper limit for scipy<=1.10 is limiting the installation which version of Python one can use for the latest version of DeepLabCut 3.0.0rc4 since scipy<=1.10 does not support the latest version of Python due to the restriction that scipy places on numpy which restricts the python version.

Is there a reason for this? Can the package limit be removed for scipy?

n-poulsen commented 1 month ago

Hi @tabedzki - we pinned the scipy version as release 1.11 made some tests fail. I don't think I can loosen the pin without rewriting some code to ensure everything runs smoothly. I'll check it out anyways and see what I can do.

Currently, we support python 3.10 and 3.11 with DeepLabCut 3.0.

tabedzki commented 1 month ago

Thank n-poulsen for getting back to me on this.

Do you remember how big of an incompatibility it was to use version 1.12? As Python 3.13 comes out this month, I'd be happy to spend some time looking into addressing the scipy limit if you have any high level remarks about where the issue was.

n-poulsen commented 1 month ago

It should be a pretty low-hanging fruit. I went back to the pull request pinning scipy (#2290), and it seems that the underlying issue has already been solved in #2396. So we should be able to un-pin scipy and work with more recent package versions.

I tested locally with python==3.12 and scipy==1.14.1 and all tests pass. Note that I also had to unpin tables==3.8.0 as it also wasn't available for python==3.12. Some of the tests need to be ignored, as you cannot install a tensorflow version that is compatible with both python==3.12 and deeplabcut. If you want to try things out locally and check that everything works for you that would be awesome

tabedzki commented 3 weeks ago

Thanks for checking that they they addressed the issues with respect to scipy so that we can remove the dependency limit.

As for pytables, we could make 3.8 the lower bound for Tables so that Python 3.12 can install the newer versions but the older one can still be used for older version of Python. Thoughts?