YaroBear / task-engine-py

MIT License
0 stars 2 forks source link

Implementation for Parallelism #10

Open pheebner opened 1 month ago

pheebner commented 1 month ago

Added ability to execute tasks in parallel. Started using node-based data structures to do this partly because I wasn't sure about how thread safety might work when modifying lists, but after having finished this it may have been OK to use lists of tasks as long as they were on the main thread. But I also wanted a way to efficiently navigate through task dependencies from the starting task through all its dependents and the node structure seemed to work well for that.

After the data structure was finished I started trying to use ThreadPoolExecutor. My first try at this worked but I don't think it was a good design. It created another thread pool executor for each node and almost certainly wouldn't scale well. v2 seems to work well though and I think it should scale much better! It only uses one ThreadPoolExecutor.

I also added a launch.json file that runs your currently open python file using the virtual environment from the examples directory. It sets the current working directory to the examples directory as well. This is good for executing example __init__.py files.

Feel free to take and leave what you'd like!