Task: Batching / Pydra Optimizations

Description

As the project develops, many of our tools work with lists of Audio objects with the goal to be that they can be optimized into Pydra workflows and have easy to use pipelines, especially for those with minimal experience. Part of this task also comes with understanding how to make this robust and easy to implement such that a user can ask for a simple task and doesn't have to be the one to consider how to batch the audios and split up the audios to optimize for Pydra (i.e. I have 64 audios and a choice between a GPU that can do batch sizes of 8 vs. 8-CPUs/cores that I can thread across).

Tasks

[ ] Pydra interface within senselab that allows for easy developmental integration (i.e. if you design a function that takes a List of Audios, we might have some logic that could easily take a reference to this function and optimize inputs to it, where our APIs would then use this logic if no user specification is given)
- [ ] Discuss whether a custom interface for this is needed or if Pydra's existing functionality is easy enough to use for our development
- [ ] If this custom interface is needed, design and implement logic for turning existing methods into tasks, splitting data across these methods for parallelization (including when and how to batch them together), and logic for making this choice for users within our existing APIs
- [ ] Benchmark how efficient different solutions are, possibly using different environments like Openmind or Google Colab. This will be important for making choices on when to split across many CPUs vs. when to use a GPU and what batch size might be optimal for a GPU
[ ] Replace existing interfaces/APIs with this logic under the hood to optimize our existing code
[ ] Make sure batching/batch size is available across our functionalities and used properly

Freeform Notes

No response

sensein / senselab