emorice / galp

Incremental distributed python runner
MIT License
0 stars 0 forks source link

Make virtual memory a normal resource #94

Closed emorice closed 4 weeks ago

emorice commented 4 weeks ago

Currently it's handled as cpus originally where, as a global switch. Instead it should go in the per task resource object. Note that will change the task def and therefore task names again, unless we can fiddle with pack to serialize it in a backward-compatible way.

emorice commented 4 weeks ago

Actually the last idea -- a pack annotation to drop the attribute completely if None -- would conflict with the idea of using tuples to serialize dataclasses since the fields are fixed. Suddenly, leaving the keys and ignoring unknown keys make quite more sense. It's technically also possible with tuples but more involved -- the new droppable keys need to be at the end and you need to drop everything until the last set key.

Probably a simpler way is to manually customize the task types namer, maybe pack shouldn't be there at all.