CrayLabs / SmartSim

SmartSim Infrastructure Library.
BSD 2-Clause "Simplified" License
219 stars 36 forks source link

Building SmartSim without ML backends #601

Closed m-kurz closed 1 month ago

m-kurz commented 1 month ago

Currently, SmartSim cannot be built correctly (i.e. with all necessary dependencies) without the ML backends. To reproduce this, build SmartSim using smart build --no_tf --no_pt and start an Orchestrator with the following Python snippet taken from the tutorials:

from smartsim import Experiment
exp = Experiment("tutorial-smartredis", launcher="local")
db = exp.create_database(db_nodes=1,port=6899,interface="lo")
exp.start(db)                         

The code will fail due to a missing RedisAI installation with the error

smartsim.error.errors.SSConfigError: RedisAI dependency not found. Build with `smart` cli or specify RAI_PATH

The reason is that after RedisAI is built using the smart tool, the resulting library is only installed to the lib folder if and only if the folder backends/ exists, which it does not if no backends are installed. Since after this step the original build folder is deleted and with it the compiled library.

This problem does not occur if any of the backends (TF, PT, ONNX) is installed. However, since they are not needed for many applications it would additional complications and effort to compile them if not necessary. Also compiling RedisAI by itself on pointing RAI_PATH to the installation also works, but poses additional effort.

To circumvent this problem this PR proposes to simply install the RedisAI library by itself if it was built.

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 67.61%. Comparing base (54755ad) to head (426f0d4).

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/CrayLabs/SmartSim/pull/601/graphs/tree.svg?width=650&height=150&src=pr&token=96HFI2F45E&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=CrayLabs)](https://app.codecov.io/gh/CrayLabs/SmartSim/pull/601?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=CrayLabs) ```diff @@ Coverage Diff @@ ## develop #601 +/- ## =========================================== - Coverage 71.53% 67.61% -3.92% =========================================== Files 78 78 Lines 6000 6000 =========================================== - Hits 4292 4057 -235 - Misses 1708 1943 +235 ``` [see 46 files with indirect coverage changes](https://app.codecov.io/gh/CrayLabs/SmartSim/pull/601/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=CrayLabs)