NDCMS / lobster

A userspace workflow management tool for harnessing non-dedicated resources for high-throughput workloads.
MIT License
3 stars 14 forks source link

Analyze Lobster with dependency tool to understand how to plan Py3 migration #656

Open klannon opened 1 year ago

klannon commented 1 year ago

For converting from Py2 to Py3, it's helpful to know if we can work in subsets. However, you can't convert c.py to Py3 if it imports a and b from a.py and b.py which are still written in Py2. There are techniques for analyzing your tree of imports/dependencies. For example, Dario Mapelli used pydeps for the WMCore py3 migration.

See: https://cms-dmwm-test.docs.cern.ch/py2py3/tools/pydeps/

(Actually, that whole site is a really great resource, but note that we don't need to make Lobster compatible with both Py2 and Py3. We can just update it to Py3 and move forward, which is a bit simpler than what Dario was working on.)

amaltaro commented 1 year ago

I don't know if it already exists somewhere else, but it would be helpful to take this opportunity and create a requirements.txt file with all the python dependencies. This will make management of dependencies easier, one could enable the GitHub feature to alert on security vulnerability and make it easier to build specific environments/images.

btovar commented 3 months ago

@mjcurran, could you take a first crack at this one? It would be helpful to know which packages in lobster_env.yaml are dependencies of other packages (i.e., not use directly by lobster).

Once we have that, we can look for either intermediate versions that both support py2 and py3, or maybe directly move to py3.

I removed already snakebite and elastic search, so it may be that some dependencies are not needed in lobster.