The current script wtables/feature_extraction/TableClassificator.py processes sequentially all the files in the folder of the compressed .tar file. Due to the large number of files and the time required for decompression, it would be good to add some parallelism.
The current approach is embarrassingly parallel and each page can be processed by a different process/thread. For testing purposes, we need a smaller sample of HTML files.
[x] Generate a smaller .tar file with only 10K samples for testing purposes.
The current script
wtables/feature_extraction/TableClassificator.py
processes sequentially all the files in the folder of the compressed.tar
file. Due to the large number of files and the time required for decompression, it would be good to add some parallelism.The current approach is embarrassingly parallel and each page can be processed by a different process/thread. For testing purposes, we need a smaller sample of HTML files.
.tar
file with only 10K samples for testing purposes.The parallelisation can be done using joblib. An example, here https://pythonhosted.org/joblib/parallel.html
I will generate such a sample and apply the parallelism over the for loop.