Parallelisation of `TableClassificator.py`

[emir-munoz]

The current script wtables/feature_extraction/TableClassificator.py processes sequentially all the files in the folder of the compressed .tar file. Due to the large number of files and the time required for decompression, it would be good to add some parallelism.

The current approach is embarrassingly parallel and each page can be processed by a different process/thread. For testing purposes, we need a smaller sample of HTML files.

• [x] Generate a smaller .tar file with only 10K samples for testing purposes.

The parallelisation can be done using joblib. An example, here https://pythonhosted.org/joblib/parallel.html

I will generate such a sample and apply the parallelism over the for loop.

[emir-munoz]

Some relevant links:

• https://gist.github.com/rahulrajaram/5934d2b786ed2c29dc418fafaa2830ad
• https://dmitryulyanov.github.io/if-you-are-lazy-to-install-slurm/
• https://stackoverflow.com/questions/36616714/return-results-to-parent-process-using-joblib

[jhomaralc]

TableClassificator.py was replaced by ExtractTables.py. The method reads article bz2 files and extracts usefull tables in HTML format.

[emir-munoz]

Thanks @jhomaralc good work. I will close this issue then.