OOM error while processing big file (150GB) - Githubissues

PlanTL-GOB-ES / corpus-cleaner

Generic toolkit for corpus cleaning

MIT License

5 stars 0 forks source link

OOM error while processing big file (150GB) #77

Open onadegibert opened 3 years ago

onadegibert commented 3 years ago

output path: /gpfs/projects/bsc88/corpus-cleaner/15-12-2020/output/oscar-de-sentence-split-2021-01-08-2114
log path: /gpfs/projects/bsc88/corpus-cleaner/15-12-2020/logs/oscar_de_corpuscleaner_13631667.err

Error: slurmstepd: error: Detected 309 oom-kill event(s) in step 13631667.batch cgroup.

Also tried to split the file into 2 batches and got the same error.