CrayLabs / SmartSim

SmartSim Infrastructure Library.
BSD 2-Clause "Simplified" License
227 stars 37 forks source link

More granular installation of SmartSim #228

Open ashao opened 2 years ago

ashao commented 2 years ago

Description

SmartSim's two-step install process (a 'pip install' followed by the smart build step) can inadvertently change versions of packges that exist in already established conda or Python virtual environments. This is particularly a problem in the smart build step where we rely on fixed versions of the ML backends for compatibility with RedisAI. While we currently allow users the option to specify a path to tensorflow and torch installations, smart build will currently attempt to install the torch package.

During the pip install phase, these ML packages are only installed if the [ml] extras is specified. The default SmartSim install thus has a minimal number of dependencies, but the documentation needs to be updated to explain the SmartSim capabilities that are enabled by the different variants of both the pip install and smart build.

Justification

Give the users guidance and control over what python packages are installed during the SmartSim install. Users will thus more likely be able to incorporate SmartSim into existing python environments and reduce the potential for dependency conflicts. This is particularly pertinent for users who may have compiled custom versions of python or ML libraries optimized for their systems and hardware.

Implementation Strategy

MattToast commented 1 year ago

With the merge of #336, SmartSim will no longer fetch ML deps within smart build, meaning there is no risk that a user may accidentally alter their existing python environment via smart.

If a python ML dep is missing or an unexpected version is detected during smart build a warning will be issued, but it will not stop the user from building the ML backends for SmartSim.

Further documentation regarding the smart build (particularly a demonstration showing the --torch_dir and --libtensorflow_dir flags) could be useful in showing users how they may build SmartSim against newer versions of Torch/Tensorflow.