search

kennyn510 / wpa2-wordlists

A collection of wordlists dictionaries for password cracking
https://kennyvn.com/wordlists-password-dictionaries-for-kali-linux/
1.23k stars 444 forks source link

Out of memory when using AWK #6

Open fa1rid opened 4 years ago

fa1rid commented 4 years ago

Hi, Using this command awk '!(count[$0]++)' old.txt > new.txt with huge dictionary is causing out of memory problem and command to be killed by the kernel.

[16346.370650] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-5.scope,task=awk,pid=17277,uid=0
[16346.370669] Out of memory: Killed process 17277 (awk) total-vm:31343560kB, anon-rss:31336728kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:61380kB oom_score_adj:0
[16347.841088] oom_reaper: reaped process 17277 (awk), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
johnchen40904 commented 4 years ago

In my opinion, the easiest way to fix this was to create a bigger swap partion (to use part of your hard drive as memory). Use gparted to resize the partions.