Finalisers blocking resource deletion

[edfincham]

Describe the bug

• The finaliser for models can prevent the removal of these resources. The workaround is to manually remove the finaliser from the resource
• This behaviour does not always occur. I have found that it most reliably occurs with model resources after something goes wrong with the model. In my current load-testing use case, the load causes the server pod to restart and reload the model
• To remove the model, I have to manually delete the finaliser. However, after that, when no model resources exist in the namespace, it still shows up in a "ModelTerminating" state when listing models via the CLI.
• When I reapply the model manifest, it does not always download the environment tarball (this may be because it still exists on the PVC? But even purging the PVC does not solve the "ModelTerminating" state)
• A similar issue can happen with servers, but I can't reliably reproduce it
• Link to Slack thread

Expected behaviour

• kubectl delete model/name to remove model resources without issue

Environment

• EKS (1.21.14)

Resources

• Nothing fancy here:

  apiVersion: mlops.seldon.io/v1alpha1
  kind: Server
  metadata:
  name: mlserver-test
  namespace: seldon-v2
  spec:
  serverConfig: mlserver
  replicas: 3
  podSpec:
  serviceAccountName: seldon-v2-sa
  containers:
  - name: mlserver
    env:
    - name: MLSERVER_PARALLEL_WORKERS
      value: "5"
  ---
  apiVersion: mlops.seldon.io/v1alpha1
  kind: Model
  metadata:
  name: test-model
  namespace: seldon-v2
  spec:
  replicas: 3
  storageUri: "s3://seldon-artifact-storage-bucket/model_version_name/model_version_number"
  secretName: "seldon-rclone-secret"
  server: mlserver-test

[kevinnowland]

I am seeing what I think is a related issue. This is all in kind, deployed with latest helm charts this morning.

Setup:

1. Deploy a server (mlserver in this case)
2. Deploy a model that gets put onto that server

Action:

1. Attempt to (mistakenly) delete server before deleting model. The server does not delete, which is expected, as it has models deployed to it.
2. Attempt to delete model to resolve failed server delete

Result:

1. Model does not delete, server does not delete

Workaround:

1. Delete model finalizer and then model deletes, but server does not yet delete
2. Delete server finalizer and then the server deletes.

Expectation: Deleting the model should actually delete the model and with all models removed, the sever delete should go through.

Please let me know if my expectation is out of line or my issue is not related to the original post. Thanks!

[edfincham]

Hey @kevinnowland, accidentally deleting a server with associated models before deleting the models themselves is definitely a scenario I have come across (and your workaround is what I've done to remedy it). However, in this case I am talking about deleting the models first :slightly_smiling_face:

[kevinnowland]

@edfincham True, that's a big difference. I just brought it up because I'd still expect the model to delete in this scenario and removing the finalizer was again a solution. But you're right, it's not caused by a failed model, there's an issue with the server first.

[edfincham]

As a follow up to this point, while you can force-delete the model resource by removing the finaliser, the model metadata persists within Seldon:

model           state           reason
-----           -----           ------
test-model      ModelTerminating

This makes it hard to reapply the model manifest since Seldon believes it is still in a terminating state. My current workaround here is to remove and reinstall seldon-core-v2-runtime, which isn't really an option for a production system.

[Kolajik]

@cliveseldon Would you happen to know when you guys are able to have a look at this issue, please? We are blocked currently because of this and cannot deploy further as it's pretty big issue for us 😞

[dtpryce]

Yeah this is blocking us on deploying to various environments, please can we get an idea of a fix and timeline please?

[dtpryce]

@cliveseldon awesome stuff getting the finalisers added for Servers and unblocking most of our environments. I know you wanted to add Models too but didn't have a chance. We would still be very interested in getting this too mostly for our development environments. Any chance of this?

[Kolajik]

I've noticed that if anything at all goes wrong with the Server onto which a Model needs to be deployed (specified in Model object via spec.server value), the finalizers on the Model object are just stuck.

I also happen to need to restart seldon-scheduler in order to correctly load the model on a Server again in any further try. There's some bad state internally in the scheduler for a Model which was not successfully loaded onto a Server.

I.e. this example we've discovered last time:

1. try to set MLSERVER_PARALLEL_WORKERS (https://mlserver.readthedocs.io/en/latest/reference/settings.html#mlserver.settings.Settings.parallel_workers) to some high number (we've had 4),
2. Have a large custom model (1+ GiB .pkl file)
3. Set MLServer's memory to 1 GiB (both request and limit)
4. Let the model (.pkl file) load via https://github.com/SeldonIO/MLServer/blob/master/mlserver/model.py#L58 method
   • this is happening automatically via agent and rclone container in SCv2
5. See if the model loads (it will not, because all of the workers cannot load such a huge .pkl file at the same time - they don't have memory available)
6. Try to delete Model object (you cannot, because of the finalizers)

So after the sixth step, I need to:

1. manually remove finalizers from Model object,
2. delete Server object,
3. restart Seldon-Scheduler,
4. redeploy Server and Model objects, in order to successfully trigger the loading of the model via agent and rclone into mlserver container.

This is all very tiring and not production ready at all.

[ramonpzg]

Hello Everyone. I just wanted to comment on this active issue to let you know that we are working on solving this and on improving model state management as well. We will have more details on the fixes in the new year.