[CK2/CM] Handle multiple python version dependencies

[arjunsuresh]

Suppose a script is having dependency on onnxruntime and pytorch and system default python is 3.10

• onnxruntime requires python <=3.9
• pytorch works for python3.10

How can we ensure that python version remains the same for all the python dependencies of the script?

[arjunsuresh]

Currently once we select a python version manually, and if some script asks for a different version, script run fails with the below error which is reasonable.

* Searching for collective script(s) with tags "get,python3"
      - Found script::get-python3,d0b5dd74373f4a62 in /home/cmuser/CM/repos/mlcommons@ck/cm-mlops/script/get-python3
        - Requested version:  >= 3.7.0   <= 3.9.999  (3.9.12)
      - Checking if script execution is already cached to skip deps ...
      - Checking if script execution is already cached ...
        - Prepared tags: -tmp,get,python3,python,get-python,get-python3
CM error: The version of the previously remembered selection for a given script (3.10.6) mismatches the newly requested one!

During the first run (before any dependency is cached) if a python dependency gets build with say python 3.8 and then another python dependency asks for python 3.9, the script run proceeds which can cause issues. Ideally we should start processing from the beginning with python3.9 here. Another option is to not allow a python version to change due to a script run, if any other script has used the current python version.

[arjunsuresh]

In particular we are facing issue in mlperf inference app because if loadgen is built using a python version say python3.7, it cannot be used by another python version. More details here

[gfursin]

As discussed, let's add "persistent_version":True to a dependency such as "python" - if we install loadgen with Python 3.8 and then get onnxruntime with forced Python 3.9, it should fail rather than trying to still pick up loadgen with Python 3.8 (can't work because of C bindings) ...

We will need to keep "persistent_cache" dict to check that a given script was picked up from cache in the workflow ...

[arjunsuresh]

@gfursin Can we add version_persistent as an option to any dependency like version, version_min etc? In this case once the version of the dependency is resolved, we can store the dependency script ID and the resolved version in a persistent object. If the same dependency script is called again with a different version, CM should throw an error.