CU-8694vv985 transitive deps

[mart-r]

Add all transitive dependencies to the environmental snapshot.

We added transitive dependencies to MedCAT in v1.12 (#438). However, that only included direct dependencies of MedCAT. In principle, the compatible versions of transitive dependencies should be handled by the package that defines them. However, these things can change over time and there may be a situation where the requirements ranges previously set no longer work properly. As such, by including all the transitive dependencies (and their versions) in the environmental snapshot could help us recreate an environment where we'd expect a model to work more confidently.

There's a few decisions I made that could very well be open to change:

1. Currently, the transitive dependencies are in a separate part of the json
   • I thought it would be good to explicitly separate them from direct dependencies
   • Though if there's good reasons to keep all dependencies together, it could be changed
2. Currently, the default behaviour includes transitive dependencies
   • It shouldn't be too computationally expensive
   • But if there's a good reason, we can have it disabled by default

The change to the default environmental snapshot as a result of the PR is as follows:

• Previously:

  {
  "dependencies": [
  ["aiofiles", "23.2.1"], ["humanfriendly", "10.0"], ["click", "8.1.7"],
  ["ipywidgets", "8.1.2"], ["psutil", "5.9.8"], ["tqdm", "4.66.4"],
  ["transformers", "4.40.2"], ["gensim", "4.3.2"], ["spacy", "3.6.1"],
  ["xxhash", "3.4.1"], ["jsonpickle", "3.0.3"], ["datasets", "2.20.0"],
  ["torch", "2.2.1"], ["pandas", "2.2.1"], ["numpy", "1.25.2"],
  ["pydantic", "1.10.14"], ["scipy", "1.9.3"], ["scikit-learn", "1.4.1.post1"],
  ["multiprocess", "0.70.16"], ["accelerate", "0.30.1"], ["peft", "0.11.0"],
  ["blis", "0.7.11"], ["dill", "0.3.8"]
  ],
  "os": "macOS-14.5-arm64-arm-64bit",
  "cpu_architecture": "arm64",
  "python_version": "3.10.13"
  }

• Now:

  {
  "dependencies": [
  ["aiofiles", "23.2.1"], ["humanfriendly", "10.0"], ["click", "8.1.7"],
  ["ipywidgets", "8.1.2"], ["psutil", "5.9.8"], ["tqdm", "4.66.4"],
  ["transformers", "4.40.2"], ["gensim", "4.3.2"], ["spacy", "3.6.1"],
  ["xxhash", "3.4.1"], ["jsonpickle", "3.0.3"], ["datasets", "2.20.0"],
  ["torch", "2.2.1"], ["pandas", "2.2.1"], ["numpy", "1.25.2"],
  ["pydantic", "1.10.14"], ["scipy", "1.9.3"], ["scikit-learn", "1.4.1.post1"],
  ["multiprocess", "0.70.16"], ["accelerate", "0.30.1"], ["peft", "0.11.0"],
  ["blis", "0.7.11"], ["dill", "0.3.8"]
  ],
  "os": "macOS-14.5-arm64-arm-64bit",
  "cpu_architecture": "arm64",
  "python_version": "3.10.13",
  "transitive_deps": [
  ["PyYAML", "6.0.1"], ["accelerate", "0.30.1"], ["aiohttp", "3.9.3"],
  ["catalogue", "2.0.10"], ["comm", "0.2.1"], ["cymem", "2.0.8"],
  ["dill", "0.3.8"], ["filelock", "3.13.1"], ["fsspec", "2023.10.0"],
  ["huggingface-hub", "0.23.0"], ["ipython", "8.22.1"],
  ["jinja2", "3.1.4"], ["joblib", "1.3.2"], ["jupyterlab-widgets", "3.0.10"],
  ["langcodes", "3.3.0"], ["multiprocess", "0.70.16"], ["murmurhash", "1.0.10"],
  ["networkx", "3.2.1"], ["numpy", "1.25.2"], ["packaging", "23.2"],
  ["pandas", "2.2.1"], ["pathy", "0.11.0"], ["preshed", "3.0.9"],
  ["psutil", "5.9.8"], ["pyarrow", "15.0.0"], ["pyarrow-hotfix", "0.6"],
  ["pydantic", "1.10.14"], ["regex", "2024.5.10"], ["requests", "2.32.3"],
  ["safetensors", "0.4.2"], ["scipy", "1.9.3"], ["setuptools", "65.5.0"],
  ["smart-open", "6.4.0"], ["spacy-legacy", "3.0.12"], ["spacy-loggers", "1.0.5"],
  ["srsly", "2.4.8"], ["sympy", "1.12"], ["thinc", "8.1.12"],
  ["threadpoolctl", "3.3.0"], ["tokenizers", "0.19.1"], ["torch", "2.2.1"],
  ["tqdm", "4.66.4"], ["traitlets", "5.14.1"], ["transformers", "4.40.2"],
  ["typer", "0.9.0"], ["typing-extensions", "4.10.0"], ["wasabi", "1.1.2"],
  ["widgetsnbextension", "4.0.10"], ["xxhash", "3.4.1"]
  ]
  }

  NOTE: The default JSON format would be different (more concise) in either case.

[tomolopolis]

Task linked: CU-8694vv985 Add more dependencies to dependency snapshot

[mart-r]

env snapshotting happens as part of model pack creation right? should be fine to include t-dep computation, as hash computation is most time consuming

Yes, that's when it happens. Not much harm, really. Even if the faster hashing is used (which nearly instant), saving the CDB on disk will take far longer than the fractions of the second finding the transitive dependencies would take. (though haven't looked at the exact timings).

Does this change integrate with the 'notification' of loading a model pack that is not the 'same' environment?

I don't think there's currently anything in MedCAT that does that. spaCy does warn when loading an older version of their model that's technically not (guaranteed to be) compatible with the current spaCy version. But I don't recall us checking the environment in any meaningful way.