artificialwisdomai / origin

Artificial Wisdom™ Cloud Platform
Apache License 2.0
2 stars 4 forks source link

Add a multiprocessing task handler #163

Closed sdake closed 3 months ago

sdake commented 3 months ago
This code enables running a collection of tasks in paralell. Initially the code runs locally using
multiprocessing. Remote targets could be added in the future.

- Integration with our theme.
- Uses fork() as part of Multiprocess().
- Uses rich for screen management.
- Tasks can have dependencies.

(v-rich) sdake@wise-a40x2:~/repos/origin/mouseion/mouseion$ python tasks.py
Artificial Wisdom™ NLP Tools • RETRO Tokenizer
• Load jsonl • ━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ • 0:00:00 • 0:00:03 •  10% •  •
• Tokenize   • ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ • 0:00:00 • 0:00:06 •   5% •  •
sdake commented 3 months ago

@rstarmer can you approve the PR so I can merge it? TY!