rfcx / arbimon-legacy

https://arbimon.org
Apache License 2.0
0 stars 0 forks source link

Switch RFM to use Kubernetes jobs #1631

Closed antonyharfield closed 1 week ago

antonyharfield commented 1 week ago

Is your feature request related to a problem? Please describe. The RFM was upgraded due to mysql library incompatibility since the database upgrade. We have migrated the job to use Kubernetes to free resources (the Arbimon jobs server) and to make it consistent with other jobs. Now we need to update the frontend and API code to start the job in Kubernetes.

Describe the solution you'd like We need to update the RFM train and classify jobs when they get created to start a Kubernetes job. The json configuration will be similar to soundscapes, with the following changes:

...
            "spec": {
                "containers": [
                    {
                        ...
                        "args": ["train_legacy"],
                        "env": [
                            {
                                "name": "JOB_ID",
                                "value": "{{ENV_JOB_ID}}"
                            }
                        ]
                    }
...

Theargs for train is train_legacy and for classify is classify_legacy. The config/secrets will be the same db and s3 config as soundscapes.

Describe alternatives you've considered None

Additional context The RFM job repo is now https://github.com/rfcx/arbimon-rfm.