Backend.AI is a streamlined, container-based computing cluster platform that hosts popular computing/ML frameworks and diverse programming languages, with pluggable heterogeneous accelerator support including CUDA GPU, ROCm GPU, TPU, IPU and other NPUs.
Although we are planning to implement a separate watcher framework, until then, we need to live with the single-module agent watcher.
It currently uses a hard-coded ipc-base-path /tmp/backend.ai/ipc, which is the default for other service daemons like storage-proxy, manager, etc.
The problem is that, after #1246, there are cases that agent + agent watcher runs together with storage-proxy, and if storage-proxy's ipc-base-path option is left as the default, storage-proxy may fail to initialize the logger socket when the agent watcher starts earlier and creates the ipc directory as the root privilege.
We need to teach the agent watcher to use a separate ipc-base-path configuration.
In the meanwhile, we could explicitly configure storage-proxy's ipc-base-path to other than the default to workaround the issue.
Although we are planning to implement a separate watcher framework, until then, we need to live with the single-module agent watcher.
It currently uses a hard-coded ipc-base-path
/tmp/backend.ai/ipc
, which is the default for other service daemons like storage-proxy, manager, etc.The problem is that, after #1246, there are cases that agent + agent watcher runs together with storage-proxy, and if storage-proxy's
ipc-base-path
option is left as the default, storage-proxy may fail to initialize the logger socket when the agent watcher starts earlier and creates the ipc directory as the root privilege.We need to teach the agent watcher to use a separate
ipc-base-path
configuration. In the meanwhile, we could explicitly configure storage-proxy'sipc-base-path
to other than the default to workaround the issue.