oschulz
commented
1 month ago CC @JBlaschke , thanks for pointing out the potential of ElasticManager to me.
Open oschulz opened 1 month ago
oschulz
commented
1 month ago CC @JBlaschke , thanks for pointing out the potential of ElasticManager to me.
Adds several things to ElasticManager:
An callback option - this can be used to automatically run init code on new workers, add them to and remove them from worker pools, add custom logging when workers connect, etc.
More debug logging - often necessary to find out what's wrong if workers won't connect.
Add a mechanism to forward environment variables to workers. Havent' found a way to set them before the Julia worker process starts up, but at least sets them before it does anything.
I'm field-testing this via a local copy of ElasticManager in ParallelProcessingTools.jl (will release a new version soon) so I can make breaking changes still if necessary, but I'll keep this PR in sync to upstream once it seems fully stable (looking pretty good so far, so hopefully soon).