search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.13k stars 1.4k forks source link

Singularity Integration #13757

Open nx10 opened 1 year ago

nx10 commented 1 year ago

What's the use case?

We are currently investigating Dagster for the use in scientific and high performance computing (HPC).

Many (shared) HPC platforms use Singularity instead of Docker. (As singularity directly integrates with the host (file) system and retains user security privileges.)

Ideas of implementation

Apart from minor differences, the Singularity CLI is very similar to Dockers.

it would be fantastic if Singularity could get Orchestration support in Dagster similar to Docker.

Additional information

Thank you for your time and consideration.

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

tacastillo commented 1 year ago

Hey Florian! Nice to hear from you again.

I'll find the right person on the team and bring this to their attention!