jkhenning
commented
1 year ago Hi @graelo,
However there's one thing I could not get to work correctly with podman: the agent-services. I can launch it and see the clearml-services worker in the services queue, tasks can be scheduled on it, but they are blocked as soon as the spun docker job needs to communicate with the API. The docker container of course does not have access to the podman networks where the services live.
I am not sure I understand access to which services does not work, and from which docker container exactly (from the agent's container? from the spun task container?) - can you please elaborate?
graelo
prassanna-ravishankar
valentinschabschneider
Hi, thanks for ClearML, it's awesome.
I orchestrate my containers on a single server using podman (managed with systemd units, with all stuff in zfs datasets). It works great. I'll share my setup, but it's basically a reformulation of the docker-compose file.
However there's one thing I could not get to work correctly with podman: the agent-services. I can launch it and see the clearml-services worker in the services queue, tasks can be scheduled on it, but they are blocked as soon as the spun docker job needs to communicate with the API. The docker container of course does not have access to the podman networks where the services live.
I tried to avoid docker in the following way: I built a new image where the services/entrypoint.sh simply starts the agent without the
--dockerflag and arguments. However, a new worker-id being added to the queue and it's not entirely clear to me but it messes up the config and the scheduled tasks fail invariably.I probably missed a simple config variable to have it working, could you point me to it? Basically, in the agent-services container, can we run
clearml-agentwithout--docker ${...}?Thanks!
PS: At the moment, I'm back to running my hpo from a non-containerized process, running the agent in pip mode, and it works great.