search

jwngr / sdow

Six Degrees of Wikipedia
https://www.sixdegreesofwikipedia.com
MIT License
1.75k stars 91 forks source link

Failed Trimming Links for 6-1-2024 wiki dump #149

Open lennonc1atwit opened 2 months ago

lennonc1atwit commented 2 months ago 
[INFO] Download date: 20240601
[INFO] Download URL: https://dumps.wikimedia.org/enwiki/20240601
[INFO] Output directory: dump

[WARN] Already downloaded sha1sums file
[WARN] Already downloaded redirects file
[WARN] Already downloaded pages file
[WARN] Already downloaded links file
[WARN] Already trimmed redirects file
[WARN] Already trimmed pages file

[INFO] Trimming links file

real    13m37.643s
user    13m43.964s
sys     3m58.226s

real    13m37.648s
user    13m43.967s
sys     3m58.229s

Tried using n1-highmem-16 (16 vCPU, 8 core, 104 GB memory) due to larger file sizes then when procedure was written (fails using recommended too), produces an empty gz temp file.

jwngr commented 2 months ago

Possibly related to https://github.com/jwngr/sdow/issues/100.