The CustomResourceDefinition "inferenceservices.serving.kserve.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

yurkoff-mv commented 11 hours ago

Validation Checklist

[X] Is this a Kubeflow issue?
[X] Are you posting in the right repository ?
[X] Did you follow the Kubeflow installation guideline ?
[X] Is the issue report properly structured and detailed with version numbers?
[X] Is this for Kubeflow development ?
[ ] Would you like to work on this issue?
[ ] You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

master

Describe your issue

When deploying using kustomize, the following warning is issued:

The CustomResourceDefinition "inferenceservices.serving.kserve.io" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

The command microk8s kubectl get pods -n kubeflow produces the following:

NAME                                                     READY   STATUS             RESTARTS        AGE
admission-webhook-deployment-5644dcc957-9c8d4            1/1     Running            0               12m
cache-server-59dfb6fcfc-rrzdv                            2/2     Running            0               12m
centraldashboard-74fc94fcf4-6bpz7                        2/2     Running            0               12m
jupyter-web-app-deployment-7dbcd448fb-cb2zd              2/2     Running            0               12m
katib-controller-7d6984668d-rgt4t                        1/1     Running            0               12m
katib-db-manager-676776f9c-qsfsp                         1/1     Running            0               12m
katib-mysql-5c9cd9b95f-d6qjl                             1/1     Running            0               12m
katib-ui-6c6fc87849-dpxj8                                2/2     Running            0               12m
kserve-controller-manager-5f8c474f97-8w4t9               2/2     Running            4 (2m29s ago)   12m
kserve-localmodel-controller-manager-6f978d76bc-nkxmq    1/2     CrashLoopBackOff   7 (86s ago)     12m
kserve-models-web-app-67f4b9dcfd-bzc7g                   2/2     Running            0               12m
kubeflow-pipelines-profile-controller-7b7b8f44f7-cfljz   1/1     Running            0               12m
metacontroller-0                                         1/1     Running            0               12m
metadata-envoy-deployment-74dbc5bdcc-x799g               1/1     Running            0               12m
metadata-grpc-deployment-8496ffb98b-9w7s6                2/2     Running            2 (12m ago)     12m
metadata-writer-7d7dfc5b8d-wzrsm                         2/2     Running            0               12m
minio-7c77bc59b8-bk5xj                                   2/2     Running            0               12m
ml-pipeline-bf9f88745-m6mpk                              2/2     Running            0               12m
ml-pipeline-persistenceagent-f97777b7f-jlzh5             2/2     Running            0               12m
ml-pipeline-scheduledworkflow-6bbc87d49-kdw5c            2/2     Running            0               12m
ml-pipeline-ui-6cf7f5d654-m2kcs                          2/2     Running            0               12m
ml-pipeline-viewer-crd-8685d84fb6-8mw4r                  2/2     Running            0               12m
ml-pipeline-visualizationserver-75b9c88599-vwkst         2/2     Running            0               12m
mysql-758cd66576-dtbc8                                   2/2     Running            0               12m
notebook-controller-deployment-6545dbccf4-flvp7          2/2     Running            1 (12m ago)     12m
profiles-deployment-5f46f7c9bb-z696s                     3/3     Running            1 (12m ago)     12m
pvcviewer-controller-manager-55f545dfc4-768sf            3/3     Running            0               12m
tensorboard-controller-deployment-546b5886c5-rqdmk       3/3     Running            0               12m
tensorboards-web-app-deployment-5bd559766d-n8r8r         2/2     Running            0               12m
training-operator-7f8bfd56f-g58qp                        1/1     Running            0               12m
volumes-web-app-deployment-5b558895d6-xml2q              2/2     Running            0               12m
workflow-controller-784cfd9c97-w9cvd                     2/2     Running            0               12m

If you look at the PODs log, you can see that there is no such resource as a InferenceService: microk8s kubectl logs -n kubeflow kserve-localmodel-controller-manager-6f978d76bc-nkxmq

{"level":"info","ts":"2024-11-14T06:51:40Z","logger":"setup","msg":"Setting up client for manager"}
{"level":"info","ts":"2024-11-14T06:51:40Z","logger":"setup","msg":"Setting up manager"}
{"level":"info","ts":"2024-11-14T06:51:40Z","logger":"setup","msg":"Registering Components."}
{"level":"info","ts":"2024-11-14T06:51:40Z","logger":"setup","msg":"Setting up KServe v1alpha1 scheme"}
{"level":"info","ts":"2024-11-14T06:51:40Z","logger":"setup","msg":"Setting up KServe v1beta1 scheme"}
{"level":"info","ts":"2024-11-14T06:51:40Z","logger":"setup","msg":"Setting up core scheme"}
{"level":"info","ts":"2024-11-14T06:51:40Z","logger":"setup","msg":"Setting up v1alpha1 LocalModel controller"}
{"level":"error","ts":"2024-11-14T06:51:40Z","logger":"setup","msg":"unable to create controller","v1alpha1Controllers":"LocalModel","error":"no matches for kind \"InferenceService\" in version \"serving.kserve.io/v1beta1\"","stacktrace":"main.main\n\t/go/src/github.com/kserve/kserve/cmd/localmodel/main.go:147\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:271"}

Steps to reproduce the issue

while ! kustomize build ~/manifests/example | microk8s kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

OS: Ubuntu 24.04
Kubernetes: MicroK8S v1.31
Kustomize: 5.2.1

Put here any screenshots or videos (optional)

No response

juliusvonkohout commented 10 hours ago

Please check out the documentation again. The installation changed slightly with server-side apply.

while ! kustomize build example | kubectl apply --server-side --force-conflicts -f -; do echo "Retrying to apply resources"; sleep 20; done

yurkoff-mv commented 4 hours ago

Yes, this solution worked! Thank you!

kubeflow / manifests