search

shashi / FileTrees.jl

Parallel computing with a tree of files metaphor
http://shashi.biz/FileTrees.jl
Other
88 stars 6 forks source link

Pass Dagger Context to exec/compute? #32

Closed DrChainsaw closed 3 years ago

DrChainsaw commented 3 years ago

With the disclaimer that I might not have understood what role the Context plays in Dagger, wouldn't it be useful to allow passing a context to exec and compute? There is a context which is passed to compute but it seems to be ignored.

My motivation is that I want to use a new Context type for dispatch inside Dagger to treat it as a WorkerPool where I can push in workers as soon as they are created and have the environment setup with @everywhere.

Discourse thread about that use case: https://discourse.julialang.org/t/adding-remote-workers-to-ongoing-dagger-job/47196/5

This can ofc be done in user code as well, but it requires duplicating basically everything in parallism.jl as you need to pass it all the way down to mapcompute as far as I understand.

I'd be happy to draft up a PR. but given my disclaimer I'd first like some feedback as to whether this is meaningful to do.

shashi commented 3 years ago

Yeah I didn't think anyone would want this this soon. I didn't want to think about it much when I was getting it to work. Feel free to make a PR!! :)