not enough resources to create a new thread in this process

[daspri]

I do not have this problem with ettercap 0.8.0 or earlier.

In 0.8.1 or git version the ncurses gui just stopped working after Viewing Connections for a while so I tried with "ettercap -T" after a while I get this error and it stops:

ERROR : 11, Resource temporarily unavailable [ettercap/src/ec_threads.c:ec_thread_new_detached:210] not enough resources to create a new thread in this process: Operation not permitted

And in "top" the VIRT mem is 3gigs (which isn't he case in 0.8.0).

I am guessing it relates to the passive resolver (resolve hostnames is not enabled)

After that error I press CTRL-C and read the log, there are 760 "[ ettercap] ec_thread_kill_all -- terminating 2536287120 [resolv_passive ..." lines.

grep -i "terminating" ettercap-0.8.1_debug.log | wc -l 760

This is a 32bit system, 2.6.27-7

[LocutusOfBorg]

can you please try with the latest git?

[daspri]

I am using latest git. Same error in git and 0.8.1.

[koeppea]

@daspri Could you please upload the debug logfile somewhere?

[koeppea]

BTW: I'm in IRC ATM

[koeppea]

Mhh I've ran ettercap in curses mode, poisening another host even with active name resolution enabled. I couldn't recognize an increase of memory usage for more that 20 minutes operation. I've tested this on a Debian Jessie 32-bit VM.

Regarding the messages in the debug log: They're normal. When ettercap is quit, it kills all threads whether they've already been destroyed or not.

Which Distribution / version have you experienced this behaviour? If possible, do you also experience this behaviour on any other host?

[daspri]

This is an old ubuntu box. Sometimes it happens in first few minutes, sometimes takes much longer. I don't know why yet. Compiled without GTK/plugins/ipv6. Just curses in Debug mode. Cleaned log (of IPs/hostnames) is here: http://pastebin.de/raw/?id=58615

[koeppea]

Thanks for the debug log. I saw in the logfile that you've called ettercap like ettercap -T -u -r c2. Is that correct? So "c2" is a packet capture file? is it possible to get it after you put it through a pcap anonymizer?

[koeppea]

As per our discussion on IRC: I understand that the PCAP can not easily handed out. However you mentioned that the OS level is quite old: Ubuntu version "8.04 or something".

I also compiled ettercap w/o GTK, plugin and IPv6 support on a current Ubuntu version. Running a poison session, the virtual memory usage of ettercap remains stable round 46 MByte.

Please update your OS version to a current version and recompile ettercap on this OS version level. This is because ettercap makes use of differnt libraries and it's features which may be different in the past.

[daspri]

I can re-create the problem by playing back my.pcap file (ettercap -T -r file) on a 32bit virtual machine of a newer ubuntu, but not a 64bit virtual machine. I am using the git version of ettercap. The VIRT ram in "top" goes up to 3gigs and ettercap freezes (the first time it displayed an ERROR: 12, Cannot allocate memory ettercap/src/ec_conbuf.c:connbuf_add:86 virtual memory exhausted ... it usually just freezes and must be CTRL-C'd)

From http://www.thoughtpolice.co.uk/vmware/ 32bit: ubuntu-server-13.04-i386.zip But the error doesn't occur on ubuntu-server-13.04-amd64.zip (the 64bit version)

Note: If I comment out the calls to "resolv_cache_insert_passive()" from ec_dns.c, the problem doesn't occur. But I can't confirm that it is related or if it just delays a memory leak from somewhere else. (But it seems to be related IMHO)

[LocutusOfBorg]

cat pastebin.de  |grep _new |wc -l
760
cat pastebin.de  |grep terminating |wc -l
757
cat pastebin.de  |grep terminated |wc -l
379

it doesn't make sense to me... shouldn't it print also "terminated"?

[LocutusOfBorg]

terminating and terminated are two following lines in the code...

[koeppea]

$ egrep "ec_thread_new.+ created" ettercap-debuglog_0.8.1.log  | wc -l
379
$ egrep "ec_thread_init.+ syncronized" ettercap-debuglog_0.8.1.log | wc -l
379
$ egrep "ec_thread_destroy.+ terminating" ettercap-debuglog_0.8.1.log | wc -l
379
$ egrep "ec_thread_destroy.+ terminated" ettercap-debuglog_0.8.1.log | wc -l
1
$ egrep "ec_thread_destroy.+ pthread_join" ettercap-debuglog_0.8.1.log | wc -l
379
$ egrep "ec_thread_kill_all.+ terminating" ettercap-debuglog_0.8.1.log | wc -l
378
$ egrep "ec_thread_kill_all.+ terminated" ettercap-debuglog_0.8.1.log | wc -l
378
$ 

The _ec_thread_destroy: pthreadjoin is misleading as the same debug message appears in _ec_thread_killall. However it seems that the call of pthread_cancel() doesn't return for all except for one thread.

[koeppea]

Could you please try if my new branch https://github.com/koeppea/ettercap/tree/fix-thread-selfdestruction fixes the issue?

[daspri]

Yes, that tree does fix the issue.

| - ec_thread_destroy(pid); | + ec_thread_exit();

[koeppea]

Mhh exactly the same picture I get when I parse a 12MByte pcap. But without the memory exhaustion. My device has only 1GByte RAM available.

$ egrep "ec_thread_new.+ created" ettercap-0.8.1_debug.log | wc -l
362
$ egrep "ec_thread_init.+ sync" ettercap-0.8.1_debug.log | wc -l
362
$ egrep "ec_thread_destroy.+ terminating" ettercap-0.8.1_debug.log | wc -l
362
$ egrep "ec_thread_destroy.+ terminated" ettercap-0.8.1_debug.log | wc -l
1
$ egrep "ec_thread_kill.+ terminating" ettercap-0.8.1_debug.log | wc -l
361
$ egrep "ec_thread_kill.+ terminated" ettercap-0.8.1_debug.log | wc -l
361
$

I ran ettercap as ettercap -Tqur test.pcap. I really have problems to reproduce the issue. Tested on a 32-bit Debian Wheezy. I launch now a 32-bit Ubuntu (current) and try to reproduce with my test.pcap.

[koeppea]

No difference. I'm afraid the answer lies somewhere in your pcap. Could you strip it down to the DNS packets and test if it fails again: tshark -Y dns -r my.pcap -w my-dns.pcap It's not logic to me that ~400 threads can consume ~3Gig memory. That would be 7,5 MByte per thread. These name resolution threads doesn't consume that much memory. In addition it woul be interessting to me if it fails as well if you run ettercap feeding my test.pcap which I made available here. Don't wonder about the content, it has been recorded in a public network.

[daspri]

Please try this bad.pcap. http://expirebox.com/download/9b07441e99af1e78e1cb216f3b7af105.html

[koeppea]

Thanks. I can now reproduce the issue.

[koeppea]

It really seem to be related to 32-bit systems. On 64-bit systems the issue can really not reproduced.

[koeppea]

OK I think I'm on the track of the issue. On Linux a new thread get's it's own stack memory. This is controlled by ulimit -s and should default to 8 MByte whether or not the memory is used.

In your DNS traffic 880 IP addresses apprear in the replies which are being put into the name cache each by it's own thread. After about 362 threads the stack size exceeds the max virtual memory (for my case 1G RAM + 2G SWAP).

By lowering the stack size to 1M (ulimit -s 1024 as root), the file is parsed w/o any issue. I do not yet have an answer why on FreeBSD 32 or 64 bit I am not able to reproduce the issue and on a Ubuntu 64 bit as well not.

However while continue understanding, I have to rethink the non-blocking name resolution.

[koeppea]

@daspri Could you please try pull request #669?

[daspri]

That patch allows the bad.pcap from above to complete, but fails on others. Example attached. It is the result of a simple perl script to resolve the hostnames (gethostbyname) listed on http://moz.com/top500

bad3.pcap http://expirebox.com/download/a10e58cb2d05a2ea05e7e1c5f08592fc.html

ettercap -Tqur bad3.pcap ERROR : 11, Resource temporarily unavailable [ettercap/src/ec_threads.c:ec_thread_new_detached:210] not enough resources to create a new thread in this process: Operation not permitted

[koeppea]

OK seems that there is no quick win....

[koeppea]

@daspri Could you please test pull request #670?

[daspri]

It works (since it isn't spawning threads anymore).

[koeppea]

Yes I actually figured out that they are not  necessary for the passive resolution.

[koeppea]

@daspri could you also please test #672. This pull completely reworks the name resolution threading including the simplification of the passive name resolution.

[daspri]

It works on my bad*.pcap files (passive name resolution).

[koeppea]

Great. I think that now #672 is the solution and implements the name resolution more properly. Thanks for making me aware of this.

[LocutusOfBorg]

:+1:

[LocutusOfBorg]

Let's close then :) feel free to merge!

[maopsv]

0.8.2 release. Same bug on 32bit Ubuntu 14.04.1 3.16.0-38-generic Crash with 3gig memory.

[koeppea]

@maopsv please try the current version from github. This issue and the related fix came in too late to make it for the 0.8.2 release. It will be included into the next release.