search

emorice / galp

Incremental distributed python runner
MIT License
0 stars 0 forks source link

Lazy load step modules. #89

Closed emorice closed 1 month ago

emorice commented 1 year ago

We should not import all the modules specified in steps when the worker is started:

  1. It's possibly slow, while we may not even need any of them.
  2. It means that imports with side effects (like blas/jax pool creation) occur very early on, before resource limits are known.
  3. Arguably, using a whitelist of modules is not a crucial feature at all and we should not even use it.
emorice commented 1 month ago

We went with point 3 above in 06ab8d2403f7fc94de9dab7240ed34c92f3b71aa