Open achimr opened 7 years ago
Ran the python profiler cProfile on the first 100 candidates from the 2015_32 en_es data collection. These are the percentages of the cumulative time for the above steps:
This was run in the AWS us-east-1 region where the CommonCrawl data is located as well.
So downloading the content does take the majority of the time, however about 44% are spent to extract the text from HTML and tokenize it. Some avenues to investigate:
After some investigation: running the code in parallel with gnu parallel doesn't work because the input file has input records (page pairs) divided over two lines - these can get separated across different input blocks with gnu parallel and the data from the separated records cannot be downloaded/aligned. So separating the downloading from the extracting/processing and making them both parallelizable seems to be the best avenue. Also to separate network-bound loads from CPU-bound processing which then can be optimized separately.
BTW - the last avenue described above to download HTML from the meta-data service is not advisable as it would make the downloading of parallel corpora dependent on the availability of the meta-data service.
It also seems unnecessary to extract text from the HTML in the WARC files as the plain text is already available in the WET files http://commoncrawl.org/the-data/get-started/
Investigated options to enable parallel downloading:
For large language pairs with about 1.2 million candidate pairs this script takes days to run. While in this case 2.4 million web pages get downloaded and processed, it would still be useful to determine where the bottle neck lies:
Example command line:
Profile code with 10s to 100s of candidate pairs.