Kopf adoptions are not working for additional worker groups

[Matt711]

Additional worker groups are not deleted when the cluster custom resource is deleted. --> Kopf adoptions are not working

Minimal Complete Verifiable Example:

from dask_kubernetes.experimental import KubeCluster

cluster = KubeCluster(name='foo')

cluster.add_worker_group(name="highmem", n_workers=2, resources={"requests": {"memory": "2Gi"}, "limits": {"memory": "64Gi"}})

cluster.close()

In this example the worker group highmem should be deleted along with the cluster, but it isn't.

Environment:

• Dask version: 2022.2.1
• Python version: 3.9.7
• Operating System: Ubuntu 18.04.6 LTS
• Install method (conda, pip, source): pip

[jacobtomlinson]

I'm not able to reproduce this locally.

Steps I took:

• Checkout main
• Create a fresh Kind cluster with kind create cluster --name pytest-kind --kubeconfig=.pytest-kind/pytest-kind/kubeconfig
• Create CRDs with kubectl apply -f dask_kubernetes/operator/customresources/
• Start development operator with kopf run -m dask_kubernetes.operator --verbose
• Open IPython and run the MCVE you shared above

Everything gets deleted correctly for me.

[Matt711]

@jacobtomlinson I don't know it would make a difference but I was running the Operator from the kube-system namespace (following the documentation). But I'll run through the steps again...

[jacobtomlinson]

Ah right, that makes more sense. It sounds like you're using the 2022.4.1 version of the operator installed via the manifest along with the latest version of dask_kubernetes.experimental.KubeCluster from main. Those likely won't be compatible.

The add_worker_group functionality is currently unreleased so you would need to run the dev version of the operator to use it.

[Matt711]

Right (palms face) of course