search

datactive / bigbang

Scientific analysis of collaborative communities
http://datactive.github.io/bigbang/
MIT License
149 stars 52 forks source link

Out of memory exits #494

Open Christovis opened 2 years ago

Christovis commented 2 years ago

3GPP scraping suddenly stopped with the displayed message being:

3GPP_TSG_SA_WG2:  47%|###############8                  | 34805/74692 [21:51:29<5481:34:13, 494.74s/it]
Killed

Upon closer interrogation I found the following reason: Out of memory: Killed process 10245 (python3) total-vm:3983576kB, anon-rss:3766216kB, file-rss:0kB, shmem-rss:4kB, UID:0 pgtables:7588kB oom_score_adj:0

We should come up with a routine that can handle these situations.