refactor dask logic: from futures to bags

impresso / impresso-pycommons

Python module with bits of code (objects, functions) highly reusable within impresso.

http://impresso-pycommons.rtfd.io/

GNU Affero General Public License v3.0

3 stars 3 forks source link

Closed mromanello closed 6 years ago

mromanello commented 6 years ago

change the way dask is used for parallel processing
instead of defining the number of futures upfront, use dask bags and let dask do this

(note to self: some template code for this to be found in the master-db s3 import)

mromanello commented 6 years ago

TODO: remove bag and map from text.helpers.rejoin_articles