search

krmaxwell / maltrieve

A tool to retrieve malware directly from the source for security researchers.
GNU General Public License v3.0
563 stars 184 forks source link

AttributeError: 'module' object has no attribute 'from_buffer' #158

Closed tysonmax20042003 closed 9 years ago

tysonmax20042003 commented 9 years ago

Hello, I have an issue that looks similar to "Problem with new filemagic? #62". after executing maltrieve it will fail. AttributeError: 'module' object has no attribute 'from_buffer'

Please help. Thank you in advance :)

python ./maltrieve.py Processing source URLs Completed source processing Downloading samples, check log for details Traceback (most recent call last): File "./maltrieve.py", line 514, in main() File "./maltrieve.py", line 503, in main if save_malware(each, cfg): File "./maltrieve.py", line 302, in save_malware mime_type = magic.from_buffer(data, mime=True) AttributeError: 'module' object has no attribute 'from_buffer'

pip freeze output: BeautifulSoup==3.2.1 Django==1.6.1 Jinja2==2.7.2 Landscape-Client==14.12 Magic-file-extensions==0.2 Mako==0.9.1 MarkupSafe==0.18 PAM==0.4.2 Pillow==2.3.0 Pyrex==0.9.8.5 SQLAlchemy==0.8.4 Twisted-Core==13.2.0 apt-xapian-index==0.45 argparse==1.2.1 beautifulsoup4==4.3.2 bottle==0.12.0 chardet==2.0.1 colorama==0.2.5 configobj==4.7.2 dnspython==1.11.1 dpkt==1.6 feedparser==5.1.3 gevent==1.0.1 greenlet==0.4.5 html5lib==0.999 libvirt-python==1.2.2 lxml==3.3.3 nose==1.3.1 openpyxl==1.7.0 pefile==1.2.9.1 pyOpenSSL==0.13 pycrypto==2.6.1 pydeep==0.2 pymongo==2.6.3 pyserial==2.6 python-apt==0.9.3.5ubuntu1 python-debian==0.1.21-nmu2ubuntu2 requests==2.2.1 six==1.5.2 ssh-import-id==3.21 urllib3==1.7.1 volatility==2.3.1 wsgiref==0.1.2 yara-python==3.3.0 zope.interface==4.0.5

Maltrieve log: cuckoo@Cuckoo-Host:~/bin/maltrieve$ cat maltrieve.log 2015-04-10 16:08:20 140695690508096 Using /home/cuckoo/bin/Malware as dump directory 2015-04-10 16:08:20 140695690508096 Starting new HTTP connection (1): support.clean-mx.de 2015-04-10 16:08:20 140695690508096 Starting new HTTP connection (1): www.malwaredomainlist.com 2015-04-10 16:08:20 140695690508096 Starting new HTTP connection (1): vxvault.siri-urz.net 2015-04-10 16:08:20 140695690508096 Starting new HTTPS connection (1): zeustracker.abuse.ch 2015-04-10 16:08:20 140695690508096 Starting new HTTP connection (1): urlquery.net 2015-04-10 16:08:20 140695690508096 Starting new HTTP connection (1): malwareurls.joxeankoret.com 2015-04-10 16:08:20 140695690508096 Starting new HTTP connection (1): malc0de.com 2015-04-10 16:08:20 140695690508096 Setting read timeout to 60 2015-04-10 16:08:21 140695690508096 Setting read timeout to 60 2015-04-10 16:08:21 140695690508096 Setting read timeout to 60 2015-04-10 16:08:21 140695690508096 "GET /hostslist/mdl.xml HTTP/1.1" 200 11043 2015-04-10 16:08:21 140695690508096 Setting read timeout to 60 2015-04-10 16:08:21 140695690508096 Setting read timeout to 60 2015-04-10 16:08:21 140695690508096 "GET /normal.txt HTTP/1.1" 200 12039 2015-04-10 16:08:21 140695690508096 "GET /rss/ HTTP/1.1" 200 None 2015-04-10 16:08:21 140695690508096 "GET /URL_List.php HTTP/1.1" 200 None 2015-04-10 16:08:21 140695690508096 Setting read timeout to 60 2015-04-10 16:08:21 140695690508096 "GET / HTTP/1.1" 200 3977 2015-04-10 16:08:21 140695690508096 Setting read timeout to 60 2015-04-10 16:08:21 140695690508096 "GET /monitor.php?urlfeed=binaries HTTP/1.1" 200 3815 2015-04-10 16:08:35 140695690508096 "GET /clean-mx/rss?scope=viruses&limit=0%2C64 HTTP/1.1" 200 7381 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): dl.pocomissus.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): cdhwps.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): 46.160.125.167 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): 124.88.67.39 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): dl.pocodoctor.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): dl.get1993desk.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): hiroba.dqx.jp.sd.rvqsm.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): bjssj0917.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): dl.desk1992get.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): 188.129.246.244 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): buhenge.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): www.acaciadeperus.com.br 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): buhenge.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): buhenge.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): bluefile.biz 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): buhenge.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): buhenge.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): win345.cn 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): img.kuping.cc 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): www.131888.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): o678.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): it-jti.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): win345.cn 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): hiroba.dqx.jp.xs.wbgrs.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): 200211.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): www.shanzhu.info 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): 46.160.125.167 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): xiazai15.869v.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): 217.24.161.60 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): back1337.info 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): dl.pocodoctor.com 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): url.52lishi.com 2015-04-10 16:08:37 140695690508096 Setting read timeout to 60 2015-04-10 16:08:37 140695690508096 Setting read timeout to 60 2015-04-10 16:08:37 140695690508096 "GET /n/13741186/FaceTime.exe?secure=1428352234_d5ae4cd0a984a31d66754544b20f2eec HTTP/1.1" 200 524856 2015-04-10 16:08:37 140695690508096 "GET /n/511a09e6-72f8-4bb9-b1ba-74af5bc06f2f/File_Downloader.exe?secure=1427911151_56c6afbd64c55a44a3d026203b408811 HTTP/1.1" 200 524856 2015-04-10 16:08:37 140695690508096 Setting read timeout to 60 2015-04-10 16:08:37 140695690508096 Setting read timeout to 60 2015-04-10 16:08:37 140695690508096 Setting read timeout to 60 2015-04-10 16:08:37 140695690508096 "GET / HTTP/1.1" 301 None 2015-04-10 16:08:37 140695690508096 Starting new HTTP connection (1): www.1788333.com 2015-04-10 16:08:37 140695690508096 Setting read timeout to 60 2015-04-10 16:08:37 140695690508096 Setting read timeout to 60 2015-04-10 16:08:37 140695690508096 Setting read timeout to 60 2015-04-10 16:08:37 140695690508096 Setting read timeout to 60 2015-04-10 16:08:37 140695690508096 "GET /arisx06.exe HTTP/1.1" 200 1579444 2015-04-10 16:08:37 140695690508096 Setting read timeout to 60 2015-04-10 16:08:37 140695690508096 "GET /home/po/zcong.exe HTTP/1.1" 200 91648 2015-04-10 16:08:37 140695690508096 "GET /n/24387532/Setup.exe?secure=1428154331_1624b924abacdb872bdfad0ba8a0f03b HTTP/1.1" 200 524856 2015-04-10 16:08:37 140695690508096 "GET /z5 HTTP/1.1" 200 617693 2015-04-10 16:08:37 140695690508096 "GET /z2 HTTP/1.1" 200 617693 2015-04-10 16:08:37 140695690508096 "GET /z1 HTTP/1.1" 200 617693 2015-04-10 16:08:37 140695690508096 Setting read timeout to 60 2015-04-10 16:08:38 140695690508096 Setting read timeout to 60 2015-04-10 16:08:38 140695690508096 "GET /aspal/jboy/bot.exe HTTP/1.1" 404 335 2015-04-10 16:08:38 140695690508096 "GET /1Q2W3E4R5T6Y7U8I9O0P1Z2X3C4V5B/down10.zol.com.cn/skycndownernew/ViDown_1.3.0.7@5403@.exe HTTP/1.1" 200 614320 2015-04-10 16:08:38 140695690508096 Setting read timeout to 60 2015-04-10 16:08:38 140695690508096 "GET /z4 HTTP/1.1" 200 617693 2015-04-10 16:08:38 140695690508096 Setting read timeout to 60 2015-04-10 16:08:38 140695690508096 "GET / HTTP/1.1" 200 11593 2015-04-10 16:08:38 140695690508096 Setting read timeout to 60 2015-04-10 16:08:38 140695690508096 Setting read timeout to 60 2015-04-10 16:08:38 140695690508096 "GET /a/q2/342.html HTTP/1.1" 200 184114 2015-04-10 16:08:38 140695690508096 "GET /z7 HTTP/1.1" 200 617693 2015-04-10 16:08:38 140695690508096 "GET /a/q2/168.html HTTP/1.1" 200 182443 2015-04-10 16:08:38 140695690508096 Setting read timeout to 60 2015-04-10 16:08:39 140695690508096 "GET / HTTP/1.1" 200 1964 2015-04-10 16:08:39 140695690508096 Setting read timeout to 60 2015-04-10 16:08:39 140695690508096 "GET /n/30813823/BBHKLKH.part2.rar.exe?secure=1428394421_50774e1d96b350fef733ce7f3a9c5ca0 HTTP/1.1" 200 524856 2015-04-10 16:08:39 140695690508096 Setting read timeout to 60 2015-04-10 16:08:39 140695690508096 "GET / HTTP/1.1" 200 183090 2015-04-10 16:08:39 140695690508096 Setting read timeout to 60 2015-04-10 16:08:39 140695690508096 Setting read timeout to 60 2015-04-10 16:08:40 140695690508096 "GET / HTTP/1.1" 200 1952 2015-04-10 16:08:40 140695690508096 "GET /bus/QunChengYuan.exe HTTP/1.1" 200 2577886 2015-04-10 16:08:40 140695690508096 Setting read timeout to 60 2015-04-10 16:08:40 140695690508096 Setting read timeout to 60 2015-04-10 16:08:40 140695690508096 "GET /n/31834469/No_Game_NoLife-_Bonus_CD_4Scan.zip.exe?secure=1428577546_7f7e978eba81e76729db9d16e599237e HTTP/1.1" 200 524856 2015-04-10 16:08:40 140695690508096 "GET / HTTP/1.1" 200 16530 2015-04-10 16:08:40 140695690508096 Setting read timeout to 60 2015-04-10 16:08:40 140695690508096 "GET /setup_435.exe HTTP/1.1" 200 2328792 2015-04-10 16:08:41 140695690508096 Setting read timeout to 60 2015-04-10 16:08:41 140695690508096 Setting read timeout to 60 2015-04-10 16:08:41 140695690508096 Setting read timeout to 60 2015-04-10 16:08:41 140695690508096 "GET /down/@54_35607.exe HTTP/1.1" 200 687872 2015-04-10 16:08:42 140695690508096 "GET /data/theme/2012zwj/12yue/12-25/20121225104331/20121225104331.exe HTTP/1.1" 200 2015512 2015-04-10 16:08:42 140695690508096 Setting read timeout to 60 2015-04-10 16:08:42 140695690508096 "GET /files/30c2cc5521bb8f10f9805f585d7848a417920.EXE HTTP/1.1" 404 323 2015-04-10 16:08:44 140695690508096 Setting read timeout to 60 2015-04-10 16:08:44 140695690508096 "GET /index.html?app=wam&ref= HTTP/1.1" 200 266 2015-04-10 16:08:51 140695690508096 "GET /account/app/svc/login.html HTTP/1.1" 500 991

krmaxwell commented 9 years ago

Can you paste the output of pip freeze?

tysonmax20042003 commented 9 years ago

Here is the output. I do have python-magic installed. I thought it was that causing the problem. Thanks for looking at the problem.

pip freeze output: BeautifulSoup==3.2.1 Django==1.6.1 Jinja2==2.7.2 Landscape-Client==14.12 Magic-file-extensions==0.2 Mako==0.9.1 MarkupSafe==0.18 PAM==0.4.2 Pillow==2.3.0 Pyrex==0.9.8.5 SQLAlchemy==0.8.4 Twisted-Core==13.2.0 apt-xapian-index==0.45 argparse==1.2.1 beautifulsoup4==4.3.2 bottle==0.12.0 chardet==2.0.1 colorama==0.2.5 configobj==4.7.2 dnspython==1.11.1 dpkt==1.6 feedparser==5.1.3 gevent==1.0.1 greenlet==0.4.5 html5lib==0.999 libvirt-python==1.2.2 lxml==3.3.3 nose==1.3.1 openpyxl==1.7.0 pefile==1.2.9.1 pyOpenSSL==0.13 pycrypto==2.6.1 pydeep==0.2 pymongo==2.6.3 pyserial==2.6 python-apt==0.9.3.5ubuntu1 python-debian==0.1.21-nmu2ubuntu2 requests==2.2.1 six==1.5.2 ssh-import-id==3.21 urllib3==1.7.1 volatility==2.3.1 wsgiref==0.1.2 yara-python==3.3.0 zope.interface==4.0.5

krmaxwell commented 9 years ago

and :fearful: I just saw you had pip freeze above. I scrolled past it too quickly, sorry!

So I don't see python-magic installed here, but Magic-file-extensions which is a different module. Can you try pip install python-magic and see if that fixes it?

tysonmax20042003 commented 9 years ago

It worked?!? I thought I had it installed. Thanks for fixing my easy answer. Oh, I have a quick question that is not related. Can I send all Malware to a Cuckoo instance by chance? I would imagine I would need to run Maltrieve then maltrievecategorizer then see if I can get Cuckoo to analyse all files in a folder or can I send it from Maltrieve?

krmaxwell commented 9 years ago

Yes, we have Cuckoo integration built-in! Just configure the URL in maltrieve.cfg. It's set to https://127.0.0.1:8090 by default.

Glad this worked for you :+1: