MP/DP support for corpora/sources/destinations

GateNLP / python-gatenlp

Python text processing, pattern matching, and NLP framework

https://gatenlp.github.io/python-gatenlp/

Apache License 2.0

63 stars 8 forks source link

MP/DP support for corpora/sources/destinations #103

Closed johann-petrak closed 3 years ago

johann-petrak commented 3 years ago

For each corpus/source/destination implementation make it inherit from traits that represent support for multiprocessing and distributed (multi-node) processing support.

johann-petrak commented 3 years ago

Ideally this should be independent of the actual mechanism used to do MP/DP and only represent the capabilities of the class, not the mechanism of how this can be done. So as an alternative to traits this could just be a method inherited from the ABC that tells about those capabilities in some way.

johann-petrak commented 3 years ago

Implemented and used traits MultiProcessingAble and DistributedProcessingAble

Fixed as of 80582fa4bd6598a0f8ee67a0b9665929eb58d0c2