Deleting failed model doesn't work in v2

[nadworny]

Describe the bug

I have a model that failed deployment. Kubectl delete is successful but the model is still visible under Models CRDs in k8s.

To reproduce

1. Train a model and log into mlflow.
2. Deploy

   apiVersion: mlops.seldon.io/v1alpha1
   kind: Model
   metadata:
   name: xxx
   spec:
   storageUri: "azureblob://xxx"
   secretName: "xxx-secret"
   requirements:
   - mlflow

3. Delete kubectl delete -f ./k8s/model_xxx.yaml -n ${NAMESPACE}

Expected behaviour

Model is completely deleted.

Environment

• Cloud Provider: local kubernetes (docker desktop)
• Kubernetes Cluster Version

  Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.2", GitCommit:"9d142434e3af351a628bffee3939e64c681afa4d", GitTreeState:"clean", BuildDate:"2022-01-19T17:27:51Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/arm64"}
  Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:29:58Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/arm64"}

• Deployed Seldon System Images: v2 using helm installation

[ukclivecox]

Which version of Seldon Core V2 are you running as Models that fail scheduling should still be deletable in latest version.

[nadworny]

@cliveseldon how can I verify that? I installed it today using the helm charts.

[nadworny]

sorry, my bad! This works on AKS (I didn't switch ctx in k9s) but it doesn't in local k8s (docker desktop). I'll update the environment info shortly.

[ukclivecox]

Can you make sure you have latest images locally as you are running on AKS as not sure what would be the difference.

[nadworny]

I did the installation using helm on Wednesday so I'm pretty sure it's latest.

[Kolajik]

This does sound like https://github.com/SeldonIO/seldon-core/issues/5043 issue