Closed seabbs closed 4 months ago
After some time looking at this and f2f discussion, I now favour using either nextflow
or airflow
as the eventually/scalable backend.
Dagger.jl
(I think, open to push back or ideas on how to combine).Thoughts @zsusswein ?
Yeah I can see the arguments for this.
I would note though that one of the main issues is not lack of cloud support in Dagger.jl
(via Distributed.j
) as it actually has quite a lot. The problem is that Azure cloud support in Julia
is poorly integrated/implemented and so doesn't work with the rest of the ecosystem.
In terms of the DAG I see the manual DAG construction you do in airflow
etc as very similar. I think the real hit we take is we move from an all Julia workflow to one with a glue language and Julia which seems like shame (but for the points you raise seems like it makes sense).
I think for now we should press on but plan to refactor as a demonstration project that we can pitch if/when there is interest or if there is interest from others (i.e @zsusswein)
A few quick thoughts:
Moving from all-Julia to multi-language is a real hit.
I strongly agree and if we could anything other than azure batch we would be able to continue all in Julia but alas. That being said I can see an argument for a standard approach to pipelining and that might as well be very fully featured (i.e some of the above options).
containerization earlier as result
Noting that containers in julia look quite trivial: https://discourse.julialang.org/t/recommended-recipe-for-deploying-a-julia-app-in-docker-with-efficient-precompilation/95591/2. I would also say that in most good pipeline tools exactly where a specific task is actually being run is abstracted and so I would lightly pushback against this or perhaps rephrase to its key to think about how the tools you are using would require you to do this
Or are you spawning a container from a shared base image per-task in your DAG? (I vote for the latter).
Currently the latter. The nice thing about Distributed.jl
is this is all abstracted so we don't need to make an internal choice about where it rruns and on what (like in the future
ecosystem)
Shall we move this to discussion?
yes
We need to decide which pipelining tool and computational backends we are going to use.
At the moment we have a partial
Dagger.jl
implementation but its unclear how we can scale this across different types of compute (i.e local, connections via ssh, slurm and Azure batch). It is also unclear what kind of pipeline tools thatDagger.jl
offers (like the task graph, progress monitoring etc). See here for issue looking into some of this: https://github.com/JuliaParallel/Dagger.jl/issues/512Another option is the
JobSchedulers.jl
andPipelines.jl
ecosystem which would be closer to a traditional command line based workflow. However it is currently unclear if these packages support non-local compute. See here for an issue on this: https://github.com/cihga39871/JobSchedulers.jl/issues/15A final alternative is to take a mixed approach of the two or to take a simpler approach that uses more of the standard base Julia tooling.
Making any decision is not time critical as we can use local compute for our current pipeline but as an example of Julia best practices we want to have a clear steer on scalable approaches for the future.