Configure `ipc_mode` or `shm_size` on container creation

[geoffrp]

For some flows that utilize multiprocessing for ML workloads, I need to run containers with increased shared memory available under /dev/shm (--shm_size) or use the host's namespace (--ipc_mode='host'). Looking through the code, I believe this would expand DockerWorkerJobConfiguration to include these two settings, which are available in Container.create.

Proposed additions:

• Add ipc_mode and shm_size to DockerWorkerJobConfiguration, both defaulting to None to preserve default behavior.

@desertaxle

[desertaxle]

Thanks for the issue @geoffrp! I think this is a great opportunity to expose more configurability for the Docker worker so that we don't need to update the code each time a new parameter is needed.

[geoffrp]

I would tend to agree, lots of arguments to play with for custom containers. Are you suggesting removing DockerWorkerJobConfiguration? I'm assuming that this construct allows the UI to know what fields may be specified if not creating through the command line?

[desertaxle]

We'll need to expand DockerWorkerJobConfiguration to include a field that accepts an arbitrary dictionary of attributes passed directly to docker-py functions. This will allow users to modify the base job template of their work pools to suit their needs without requiring additional code changes to the DockerWorker.

[geoffrp]

@desertaxle Is there a prospective timeline for this feature? I would be willing to assist if possible