symforce-org / symforce

Fast symbolic computation, code generation, and nonlinear optimization for robotics
https://symforce.org
Apache License 2.0
1.41k stars 145 forks source link

Add versioned dev_requirements for py 3.8 3.9 3.10 #229

Open bradley-solliday-skydio opened 2 years ago

bradley-solliday-skydio commented 2 years ago

Different version of python have different dev requirements. Previously, dev_requirements.txt was generated for python3.8 and assumed to work for other versions of python.

That was more or less a safe assumption, except is now causing problems. Specifically, gtsam is not available on python 3.10, but we'd like to add it as a dev requirement for python 3.8 and 3.9.

symforce_requirements_test.py cannot be run until symforce has been installed, but one might reasonably want to pip install -r dev_requirements.txt before installing symforce (to use the pinned build depencies during installation).

To make this circular dependency less painful for the user, I'm adding

Modified test/symforce_requirements_test.py to only test/update the dev_requirements_XY.txt of the python version being used.

Since the public CI is already running this test, it will guarentee that none of these files will go out of date.

Needed to amend both ci.yml and docs.yml to use the correct dev_requirements_XY.txt.

Also needed to update the README.md accordingly.

aaron-skydio commented 2 years ago

The alternative would be to mark the requirement like gtsam ; python_version < '3.10' in setup.py, which piptools understands correctly:

diff --git a/dev_requirements.txt b/dev_requirements.txt
index ae5b80a0a9d..c61f046c924 100644
--- a/dev_requirements.txt
+++ b/dev_requirements.txt
@@ -78,6 +78,8 @@ fonttools==4.33.3
     # via matplotlib
 graphviz==0.20
     # via symforce (setup.py)
+gtsam==4.1.1 ; python_version < "3.10"
+    # via symforce (setup.py)
 idna==3.3
     # via requests
 imagesize==1.3.0
@@ -184,6 +186,7 @@ numba==0.56.0
     # via symforce (setup.py)
 numpy==1.22.4
     # via
+    #   gtsam
     #   matplotlib
     #   numba
     #   pandas
@@ -242,6 +245,7 @@ pylint==2.14.0
     # via symforce (setup.py)
 pyparsing==3.0.9
     # via
+    #   gtsam
     #   matplotlib
     #   packaging
 pyrsistent==0.18.1
diff --git a/setup.py b/setup.py
index f752b5b4eb5..1d621a45209 100644
--- a/setup.py
+++ b/setup.py
@@ -395,6 +395,7 @@ setup(
             "types-pygments",
             "types-requests",
             "types-setuptools",
+            "gtsam ; python_version < '3.10'"
         ],
         "_setup": setup_requirements,
     },

We should specify a dependency on gtsam in setup.py this way regardless - we may want to split compiled requirements for different versions of python anyway, but I don't think we need it for gtsam specifically?

bradley-solliday-skydio commented 2 years ago

Huh. I didn't realize that was possible. Well, I really only cared about gtsam. If we have time to kill, we could get this merged (perhaps by making it easier to update the dev_requirements from just one test using containers or something). But I'm content to just let this get shelved (and perhaps picked up at another time or used as reference if that'd be useful when the time comes).

aaron-skydio commented 2 years ago

Yeah I don't particularly think we need to do this now