This PR refactors the cs_workers.models.executors package into its own lightweight package cs_jobs with minimal dependencies (httpx and cs-storage). This package provides a minimal CLI that is used to start jobs from the Kubernetes api:
It also provides a module job.py for calling the appropriate function (e.g. get_inputs) and task_wrapper.py which is responsible for handling run-time failures and pushing results back to other services.
The main advantage of this is that unnecessary packages that are required to run the compute cluster like sqlalchemy, fastapi, and pydantic do not need to be installed solely to prevent import errors form elsewhere in the cs_workers package. This will also make updating models easier in the future which hopefully only happens very, very rarely.
This PR refactors the
cs_workers.models.executorspackage into its own lightweight packagecs_jobswith minimal dependencies (httpxandcs-storage). This package provides a minimal CLI that is used to start jobs from the Kubernetes api:https://github.com/compute-tooling/compute-studio/blob/54b45fd412cff9643cfc15cf09845bbae2b7ae62/workers/cs_workers/models/clients/job.py#L106-L112
It also provides a module
job.pyfor calling the appropriate function (e.g.get_inputs) andtask_wrapper.pywhich is responsible for handling run-time failures and pushing results back to other services.The main advantage of this is that unnecessary packages that are required to run the compute cluster like
sqlalchemy,fastapi, andpydanticdo not need to be installed solely to prevent import errors form elsewhere in thecs_workerspackage. This will also make updating models easier in the future which hopefully only happens very, very rarely.