search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.71k stars 657 forks source link

Singularity Support #1079

Open AkideLiu opened 1 year ago

AkideLiu commented 1 year ago

Proposal Summary

The proposed feature is to provide integrated support for Singularity in ClearML. Singularity is a powerful tool for packaging and running reproducible computational workloads. The aim is to allow users to create, manage, and execute Singularity containers directly through ClearML, just like they would with Docker. It includes being able to specify Singularity images in the ClearML configuration files and the system automatically handling the creation, management and execution of Singularity containers.

Motivation

ClearML already supports Docker for managing containers. However, in many academic, scientific, and high-performance computing (HPC) contexts, Singularity is more prevalent. It provides several features that are important in these environments, like compatibility with shared file systems, user privileges control and the ability to run containers without root access.

Adding Singularity support to ClearML would help to simplify the computational workflows of these users. They would be able to easily package their experiments in Singularity containers, run them via ClearML, and have experiment tracking and resource monitoring fully integrated.

Moreover, for organizations that have restrictions on Docker use due to security considerations, Singularity provides an excellent alternative for containerization. It would be beneficial to offer these users the flexibility of choosing the containerization technology that best suits their needs and organizational policies.

Related Discussion

ainoam commented 1 year ago

Thanks for suggesting @AkideLiu

While this is not in our immediate plans, we'll definitely add this as a potential roadmap item if ClearML's user community provides a strong demand.

Dario-Mantegazza commented 1 month ago

Any news about Singularity/Apptainer support?

ainoam commented 4 weeks ago

Thanks for showing interest @Dario-Mantegazza. This is still part of the longer term roadmap.

Dario-Mantegazza commented 4 weeks ago

Ok, if I can share my 2cents as post-doc researcher and entrepreneur is that Singularity/Apptainer is becoming more and more used in both academia and companies. In particular those that do AI. Having Clearml work with slurm and singularity would mean having a mega setup for handling AI computational jobs on shared workstations. I hope you will anticipate it in your roadmap :)