substratusai / runbooks

Finetune LLMs on K8s by using Runbooks
https://www.substratus.ai
Other
168 stars 14 forks source link

Long model names cause too long job names #256

Open samos123 opened 11 months ago

samos123 commented 11 months ago

Steps to reproduce:

  1. Create a model with following name: wgqlg-withretrieval-schemasplit-train-80-mistral-instruct

Current result: No job gets created and following log is observed:

2023-10-15T03:59:28Z    ERROR   Reconciler error        {"controller": "model", "controllerGroup": "substratus.ai", "controller
Kind": "Model", "Model": {"name":"wgqlg-withretrieval-schemasplit-train-80-mistral-instruct","namespace":"default"}, "namespace
": "default", "name": "wgqlg-withretrieval-schemasplit-train-80-mistral-instruct", "reconcileID": "fbf39151-3ddb-400f-8704-401b
3d605bb5", "error": "creating Job: Job.batch \"wgqlg-withretrieval-schemasplit-train-80-mistral-instruct-modeller\" is invalid:
 [metadata.labels: Invalid value: \"wgqlg-withretrieval-schemasplit-train-80-mistral-instruct-modeller\": must be no more than 63 characters, spec.template.labels: Invalid value: \"wgqlg-withretrieval-schemasplit-train-80-mistral-instruct-modeller\": must be no more than 63 characters]"}                                                                                             
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                                                  /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324                           sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265                           sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                                                     /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226