Refactor `inbatch_parallel`

analysiscenter / batchflow

BatchFlow helps you conveniently work with random or sequential batches of your data and define data processing and machine learning workflows even for datasets that do not fit into memory.

https://analysiscenter.github.io/batchflow/

Apache License 2.0

201 stars 46 forks source link

Refactor `inbatch_parallel` #672

Open SergeyTsimfer opened 2 years ago

SergeyTsimfer commented 2 years ago

As the inbatch_parallel now not supposed to be used on its own, we can refactor it with following goals in mind:

[ ] remove _use_self args
[ ] remove init/post functions: the container with init should be passed directly from Batch.apply_parallel, and the results should be post-processed in the Batch.apply_parallel as well
[ ] make inbatch_parallel a class: that would allow for easier introspection and parameter changes on the fly, for example, target to any other.

SergeyTsimfer commented 2 years ago

use inspect to check whether src/dst should be passed to the method

roman-kh commented 2 years ago

Avoid inspect if possible. Use inbatch_parallel arguments instead.