search

mapsacosta / htcdaskgateway

A Dask Gateway client extension for heterogeneous cluster mode combining the Kubernetes backend for pain-free scheduler networking, with COFFEA-powered HTCondor workers
2 stars 2 forks source link

Worker image is hardcoded in cluster.py #1

Open mapsacosta opened 1 year ago

mapsacosta commented 1 year ago

We are currently just assuming that the latest coffea-dask image is what the user wants, this is no longer the case as we need to be able to adjust it as needed.

A breaking change upgrading to python3.10 in docker-coffea-dask motivated this use-case.

https://github.com/mapsacosta/htcdaskgateway/blob/main/src/htcdaskgateway/cluster.py#L70 <--- this is where the image is hardcoded

It would be ideal for someone to go:

gateway = HTCGateway()
cluster = gateway.new_cluster(image_name=f"/cvmfs/unpacked.cern.ch/registry.hub.docker.com/coffeateam/coffea-dask-cc7-gateway:<your_version")
mapsacosta commented 6 months ago

@Nanoemc let's use this issue to follow-up on the testing for this feature

nsmith- commented 1 month ago

A similar problem still exists, where the image prefix is hardcoded in: https://github.com/mapsacosta/htcdaskgateway/blob/8e01209dab614ac00d265abdd14dc64706dec040/src/htcdaskgateway/cluster.py#L84 It would be nice to have the option to specify the full path cc @dalfonso