The current implementation of example applications in the repository runs exclusively on GPU. While this is necessary for certain applications like chat_with_video or llama2, other applications such as whisper can run effectively on a CPU, albeit at a slower speed. The existing deployment configuration, found in deployments.py, mandates specifying the number of GPUs (num_gpus) for all deployments. This limitation necessitates code modification to run applications on a CPU, which is not ideal. This enhancement proposes adding a feature to override the num_gpus setting without altering the code.
Advantages
Allows users to run applications on either CPU or GPU without modifying the existing codebase.
Possible Implementation
Introduce a flag or an environment variable that, when set, automatically removes the num_gpus configuration. This could be implemented as follows:
Environment Variable: Create an environment variable, e.g., USE_CPU_ONLY. When this variable is set, the deployment script should remove the num_gpus setting.
Configuration Flag: Add a configuration flag, e.g., use_cpu_only. When this flag is enabled, the deployment logic should remove the num_gpus parameter.
By implementing this enhancement, the deployment process will become more adaptable, catering to a wider range of hardware configurations without necessitating direct code changes.
Enhancement Description
The current implementation of example applications in the repository runs exclusively on GPU. While this is necessary for certain applications like
chat_with_videoorllama2, other applications such aswhispercan run effectively on a CPU, albeit at a slower speed. The existing deployment configuration, found in deployments.py, mandates specifying the number of GPUs (num_gpus) for all deployments. This limitation necessitates code modification to run applications on a CPU, which is not ideal. This enhancement proposes adding a feature to override thenum_gpussetting without altering the code.Advantages
Allows users to run applications on either CPU or GPU without modifying the existing codebase.
Possible Implementation
num_gpusconfiguration. This could be implemented as follows:USE_CPU_ONLY. When this variable is set, the deployment script should remove thenum_gpussetting.use_cpu_only. When this flag is enabled, the deployment logic should remove thenum_gpusparameter.By implementing this enhancement, the deployment process will become more adaptable, catering to a wider range of hardware configurations without necessitating direct code changes.