search

krmaxwell / maltrieve

A tool to retrieve malware directly from the source for security researchers.
GNU General Public License v3.0
562 stars 184 forks source link

Too many open files #66

Closed GelosSnake closed 9 years ago

GelosSnake commented 9 years ago

IOError: [Errno 24] Too many open files: 'urls.json

It seems that the process is running too many simultaneous file downloads? This can be fixed with extending /etc/security/limits.conf to 5000 current is: $ ulimit -n = 1000 maybe queuing system or making sure files are closed after download?

krmaxwell commented 9 years ago

Do you have any repro steps? Try as I might, I can't make this happen.

GelosSnake commented 9 years ago

I did try to reproduce and failed, I think we should close it and than maybe later open if it returns.

krmaxwell commented 9 years ago

Sounds good.

GelosSnake commented 9 years ago

Error just came back: Traceback (most recent call last): File "maltrieve.py", line 336, in File "maltrieve.py", line 326, in main IOError: [Errno 24] Too many open files: 'urls.json'

webstergd commented 9 years ago

I also just got this error today. This was from a fresh install and occurred on the first and second attempt to execute the program. First quit at ~644 files and the second quit around 1300ish. The third try seemed to work fine. I looked through the code and could not find anything that looked like it could cause this error.

ghost commented 9 years ago

You can scope limit to the process level within Python

import resource resource.setrlimit(resource.RLIMIT_NOFILE, (65536, 65536))

https://docs.python.org/2/library/resource.html

dray0n commented 9 years ago

I was having the same issues on a fresh install. I was able to remedy the issue:

$ ulimit -a open files (-n) 1024

$ ulimit -n 2048

Maltrieve has now been running for 30 minutes without an issue. Before, I was erroring out after about 5-10 minutes of running.

krmaxwell commented 9 years ago

Can one of you who's experienced the issue test the rlimit branch? It implements the fix suggested by @alienone with a limit of 2048 descriptors as @dray0n is using.

dray0n commented 9 years ago

I am going to load the rlimit branch right now.... but as a note I received the error again. Roughly 45-minutes in.

krmaxwell commented 9 years ago

This is so odd because the Python context manager (with) should close the file when exiting that block. It makes me wonder if grequests is holding something open.

dray0n commented 9 years ago

$ netstat -an

Hundreds of "CLOSE_WAIT"'s out there.

dray0n commented 9 years ago

Been awhile... I was going through the dependencies and found out that I did not have all of the requirements met for some reason. Problem solved through a few issues on missing packages and was finally able to get all requirements met.

Open socket issue is no longer an issue.

GelosSnake commented 9 years ago

cheers

krmaxwell commented 9 years ago

:+1:

jebelingmi commented 9 years ago

just got same issue reported above on fresh install today. jeff@ubuntu64:~/maltrieve$ python /home/jeff/maltrieve/maltrieve.py -l logfile.txt -p http://192.168.11.122:9090 -d /home/jeff/maltrieve/malware/ External sites see 68.40.253.188

Processing source URLs Completed source processing Downloading samples, check log for details Traceback (most recent call last): File "/home/jeff/maltrieve/maltrieve.py", line 514, in File "/home/jeff/maltrieve/maltrieve.py", line 503, in main File "/home/jeff/maltrieve/maltrieve.py", line 340, in save_malware IOError: [Errno 24] Too many open files: '/home/jeff/maltrieve/malware/bbc0dacb709608260751692405972b7b'