Open ashao opened 2 years ago
With the merge of #336, SmartSim will no longer fetch ML deps within smart build
, meaning there is no risk that a user may accidentally alter their existing python environment via smart
.
If a python ML dep is missing or an unexpected version is detected during smart build
a warning will be issued, but it will not stop the user from building the ML backends for SmartSim.
Further documentation regarding the smart build
(particularly a demonstration showing the --torch_dir
and --libtensorflow_dir
flags) could be useful in showing users how they may build SmartSim against newer versions of Torch/Tensorflow.
Description
SmartSim's two-step install process (a 'pip install' followed by the
smart build
step) can inadvertently change versions of packges that exist in already established conda or Python virtual environments. This is particularly a problem in thesmart build
step where we rely on fixed versions of the ML backends for compatibility with RedisAI. While we currently allow users the option to specify a path to tensorflow and torch installations,smart build
will currently attempt to install the torch package.During the pip install phase, these ML packages are only installed if the
[ml]
extras is specified. The default SmartSim install thus has a minimal number of dependencies, but the documentation needs to be updated to explain the SmartSim capabilities that are enabled by the different variants of both the pip install and smart build.Justification
Give the users guidance and control over what python packages are installed during the SmartSim install. Users will thus more likely be able to incorporate SmartSim into existing python environments and reduce the potential for dependency conflicts. This is particularly pertinent for users who may have compiled custom versions of python or ML libraries optimized for their systems and hardware.
Implementation Strategy
smart build
option to never install Python packages, but issue a warning that this may lead to undesirable behaviorpip install
phasepip install smartsim
will allow users to launch experiments and the orchestrator, but not allow for ML models to be loaded or run from the database)