search

krmaxwell / maltrieve

A tool to retrieve malware directly from the source for security researchers.
GNU General Public License v3.0
563 stars 184 forks source link

Error "NoneType" #26

Closed IntimacyGel closed 11 years ago

IntimacyGel commented 11 years ago

Hello Good sir!

I've been using this flawlessly for a couple weeks now. I used to have an issue where it would freeze due to all the threads getting hung on a download issue or whatever so I changed the threads to 500 and haven't heard a peep from it. However, Today it's been "finishing" prematurely at this exact same spot - URL 569. I'm really not the expert here, but this specific line references issues with not knowing the type on some file from the mcbrtext feed. I tried commenting out the entire mcbrtext feed and it froze again at URL 569 referencing cleanmxtext. I commented that feed out and it and then it just freezes at URL 569. I'm not sure if this is just some file that both of these feeds are hosting that this crawler doesn't like, but if that's the case then this issues shouldn't last long as I'm sure they'll take it out of their feed after a few days.

Thanks for your help in advance! Gel

2013-10-08 10:14:56 4036 Adding new URL to queue: http://1380677010.keaitz.com/c hat/bosom/lb_bosom_6.exe 2013-10-08 10:14:57 4036 urlopen() returned error Gone

Traceback (most recent call last): File "F:\Crawler\maltrieve.py", line 269, in main() File "F:\Crawler\maltrieve.py", line 246, in main for url in mcbrtext.read().splitlines(): AttributeError: 'NoneType' object has no attribute 'read'

F:\Crawler>_

IntimacyGel commented 11 years ago

I found that the mcbrtext xml feed is down so I guess this needs to get pulled out of the script:

mcbrtext=get_URL('http://www.malware.com.br/cgi/submit?action=list') for url in mcbrtext.read().splitlines(): if re.match('^http', url): push_malware_URL(url,malq)"

krmaxwell commented 11 years ago

It looks like one of my try/except clauses needs a fix, thanks. And I'll look into the Malware Patrol URL!

IntimacyGel commented 11 years ago

Wow man you're quick :+1: