whenever i run downlod_common_crawl.py code in examples folder after it downloaded the shards, it starts to extract the data. in between warnings come up which says this code doesnt free the memory and after a while it kills the process.
next problem is this extraction takes so long time for me, it extracts 10 shards in about 1 and 45 minutes. is there any extra configuration that i have missed?
Describe the bug
whenever i run downlod_common_crawl.py code in examples folder after it downloaded the shards, it starts to extract the data. in between warnings come up which says this code doesnt free the memory and after a while it kills the process.
next problem is this extraction takes so long time for me, it extracts 10 shards in about 1 and 45 minutes. is there any extra configuration that i have missed?
here is the SS of problem
Environment overview
Environment details