search

JuliaLang / Distributed.jl

Create and control multiple Julia processes remotely for distributed computing. Ships as a Julia stdlib.
https://docs.julialang.org/en/v1/stdlib/Distributed/
MIT License
20 stars 8 forks source link

Distributed.jl - possibility to use other Serialization libraries? #1

Open schlichtanders opened 9 months ago

schlichtanders commented 9 months ago

Hi, since recently Pluto can run Distributed, however it does not work out of the box because the standard serialization is known not to work in Pluto, but Distributed relies on it (I couldn't find a way to change this)

The PlutoSerialization is maintained in this repo https://github.com/MichaelHatherly/PlutoSerialization.jl

It would be great to be able to use this Serialization instead of the default Serialization when using Distributed inside a Pluto notebook.

jpsamaroo commented 9 months ago

ClusterSerializer could have a hook for customizing name binding lookups. I'm not sure what such a design would look like in the face of multiple libraries wanting to hook this functionality. Probably a hook could be defined per-Module, so that this hook will only be called on modules that Pluto creates.

schlichtanders commented 9 months ago

Looking into which reasoning was taken for the global random number generator it seems that the preferred route is to make the ClusterSerializer pluggable. https://github.com/JuliaLang/julia/issues/23199#issuecomment-858486131

These are the two places where the ClusterSerializer is constructed

So I guess we would need:

schlichtanders commented 9 months ago

@MichaelHatherly you build PlutoSerialization.jl, probably you have some valuable insights what could be potential solutions/problems

MichaelHatherly commented 9 months ago

Thanks for pointing this out @schlichtanders. I've got no extra insights at the moment since I've not had a chance yet to look into how Malt changes things now that Pluto is using it instead. If I get a chance I'll update this thread.

jpsamaroo commented 9 months ago

Malt shouldn't be relevant here since this issue is relating to running code under Distributed directly (IIRC Pluto only runs its own internal code under Malt)

schlichtanders commented 5 months ago

I would really love to see Distributed be fully supported inside Pluto and would like to try out things myself. (e.g. creating the pluggable ClusterSerializer interface thought about above)

@jpsamaroo do you know whether and how Distributed.jl could already be replace by a fork or the like?