Issues with volatility, httpreplay, and m2crypto

[fez219]

Hello,

I just got CUckoo more or less working and reporting, there are just a few things left I'm not sure how to fix.

No matter what I do, I get errors from httpreplay, m2crypto, and volatility. I've looked all over and tried all kinds of troubleshooting (for instance, tried everything from https://github.com/cuckoosandbox/cuckoo/issues/929 and related issues with no success) but can't fix the problems.

Here's what happens when I run a sample:

2016-08-23 13:43:06,546 [lib.cuckoo.core.scheduler] INFO: Starting analysis of FILE "npp.6.9.2.Installer.exe" (task #15, options "") 2016-08-23 13:43:06,787 [lib.cuckoo.core.scheduler] INFO: File already exists at "/home/alexander/cuckoo/storage/binaries/a329d076a349c9bffa02bfe670e1d9beaa21ce10a695db2d675e9346b6e72fca" 2016-08-23 13:43:06,887 [lib.cuckoo.core.scheduler] INFO: Task #15: acquired machine windows7-cuckoo (label=windows7-cuckoo) 2016-08-23 13:43:06,946 [modules.auxiliary.sniffer] INFO: Started sniffer with PID 5084 (interface=vboxnet0, host=192.168.56.101, pcap=/home/alexander/cuckoo/storage/analyses/15/dump.pcap) tcpdump: listening on vboxnet0, link-type EN10MB (Ethernet), capture size 262144 bytes 2016-08-23 13:43:16,230 [lib.cuckoo.core.guest] INFO: Starting analysis on guest (id=windows7-cuckoo, ip=192.168.56.101) 2016-08-23 13:45:29,208 [lib.cuckoo.core.guest] INFO: windows7-cuckoo: analysis completed successfully 187 packets captured 187 packets received by filter 0 packets dropped by kernel 2016-08-23 13:45:33,333 [modules.machinery.virtualbox] INFO: Successfully generated memory dump for virtual machine with label windows7-cuckoo to path /home/alexander/cuckoo/storage/analyses/15/memory.dmp 2016-08-23 13:45:54,706 [modules.processing.memory] ERROR: Generic error executing volatility Traceback (most recent call last): File "/home/alexander/cuckoo/modules/processing/memory.py", line 1022, in run results = VolatilityManager(self.memory_path).run() File "/home/alexander/cuckoo/modules/processing/memory.py", line 942, in run vol = VolatilityAPI(self.memfile, self.osprofile) File "/home/alexander/cuckoo/modules/processing/memory.py", line 53, in init self.init_config() File "/home/alexander/cuckoo/modules/processing/memory.py", line 109, in init_config if self.get_dtb(): File "/home/alexander/cuckoo/modules/processing/memory.py", line 59, in get_dtb for ep in ps.calculate(): File "/usr/local/lib/python2.7/dist-packages/volatility-2.3.1-py2.7.egg/volatility/plugins/filescan.py", line 424, in calculate kernel_as = utils.load_as(self._config) File "/usr/local/lib/python2.7/dist-packages/volatility-2.3.1-py2.7.egg/volatility/utils.py", line 66, in load_as raise error AddrSpaceError: No suitable address space mapping found Tried to open image as: MachOAddressSpace: mac: need base LimeAddressSpace: lime: need base WindowsHiberFileSpace32: No base Address Space WindowsCrashDumpSpace64: No base Address Space HPAKAddressSpace: No base Address Space VirtualBoxCoreDumpElf64: No base Address Space VMWareSnapshotFile: No base Address Space WindowsCrashDumpSpace32: No base Address Space AMD64PagedMemory: No base Address Space IA32PagedMemoryPae: No base Address Space IA32PagedMemory: No base Address Space MachOAddressSpace: MachO Header signature invalid LimeAddressSpace: Invalid Lime header signature WindowsHiberFileSpace32: No xpress signature found WindowsCrashDumpSpace64: Header signature invalid HPAKAddressSpace: Invalid magic found VirtualBoxCoreDumpElf64: Unknown VBox core format version VMWareSnapshotFile: Invalid VMware signature: 0x464c457f WindowsCrashDumpSpace32: Header signature invalid AMD64PagedMemory: Incompatible profile WinXPSP2x86 selected IA32PagedMemoryPae: No valid DTB found IA32PagedMemory: No valid DTB found FileAddressSpace: Must be first Address Space ArmAddressSpace: No valid DTB found

2016-08-23 13:45:55,830 [modules.processing.static] CRITICAL: You do not have the m2crypto library installed preventing certificate extraction: pip install m2crypto 2016-08-23 13:45:56,652 [modules.processing.network] ERROR: Error running httpreplay-based PCAP analysis Traceback (most recent call last): File "/home/alexander/cuckoo/modules/processing/network.py", line 764, in run results.update(Pcap2(pcap_path, self.get_tlsmaster()).run()) File "/home/alexander/cuckoo/modules/processing/network.py", line 695, in run r = httpreplay.reader.PcapReader(self.pcap_path) File "/usr/local/lib/python2.7/dist-packages/httpreplay/reader.py", line 25, in init self.pcap = dpkt.pcap.Reader(fp) File "/usr/local/lib/python2.7/dist-packages/dpkt/pcap.py", line 115, in init buf = self.f.read(FileHdr.__hdr_len) AttributeError: 'str' object has no attribute 'read' 2016-08-23 13:46:03,723 [lib.cuckoo.core.scheduler] INFO: Task #15: reports generation completed (path=/home/alexander/cuckoo/storage/analyses/15) 2016-08-23 13:46:03,814 [lib.cuckoo.core.scheduler] INFO: Task #15: analysis procedure completed

Thanks for the help!

[fez219]

In case it matters, my Volatility and httpreplay installations are up-to-date.

[doomedraven]

@fez219 your vol is not up to date, your vol is volatility-2.3.1, current is 2.5, in your version i saw in past this error, update and let me know if it works

[doomedraven]

also as errors says, you have missed library, install it with sudo pip install m2crypto

[fez219]

@doomedraven I wish it was that simple! I just uninstalled and reinstalled volatility, same error. And I have m2crypto installed. I ran the command just to be sure and got the "requirement already satisfied" spiel.

I ran another analysis, here's the results (seems to be pretty much the same thing):

2016-08-30 12:17:39,218 [lib.cuckoo.core.scheduler] INFO: Starting analysis of FILE "npp.6.9.2.Installer.exe" (task #21, options "") 2016-08-30 12:17:39,467 [lib.cuckoo.core.scheduler] INFO: File already exists at "/home/alexander/cuckoo/storage/binaries/a329d076a349c9bffa02bfe670e1d9beaa21ce10a695db2d675e9346b6e72fca" 2016-08-30 12:17:39,612 [lib.cuckoo.core.scheduler] INFO: Task #21: acquired machine windows7-cuckoo (label=windows7-cuckoo) 2016-08-30 12:17:39,634 [modules.auxiliary.sniffer] INFO: Started sniffer with PID 18497 (interface=vboxnet0, host=192.168.56.101, pcap=/home/alexander/cuckoo/storage/analyses/21/dump.pcap) tcpdump: listening on vboxnet0, link-type EN10MB (Ethernet), capture size 262144 bytes 2016-08-30 12:17:47,656 [lib.cuckoo.core.guest] INFO: Starting analysis on guest (id=windows7-cuckoo, ip=192.168.56.101) 2016-08-30 12:20:11,623 [lib.cuckoo.core.guest] INFO: windows7-cuckoo: analysis completed successfully 191 packets captured 191 packets received by filter 0 packets dropped by kernel 2016-08-30 12:20:17,762 [modules.machinery.virtualbox] INFO: Successfully generated memory dump for virtual machine with label windows7-cuckoo to path /home/alexander/cuckoo/storage/analyses/21/memory.dmp 2016-08-30 12:21:08,222 [modules.processing.memory] ERROR: Generic error executing volatility Traceback (most recent call last): File "/home/alexander/cuckoo/modules/processing/memory.py", line 1022, in run results = VolatilityManager(self.memory_path).run() File "/home/alexander/cuckoo/modules/processing/memory.py", line 942, in run vol = VolatilityAPI(self.memfile, self.osprofile) File "/home/alexander/cuckoo/modules/processing/memory.py", line 53, in init self.init_config() File "/home/alexander/cuckoo/modules/processing/memory.py", line 107, in init_config self.addr_space = utils.load_as(self.config) File "/usr/local/lib/python2.7/dist-packages/volatility-2.5-py2.7.egg/volatility/utils.py", line 65, in load_as raise error AddrSpaceError: No suitable address space mapping found Tried to open image as: MachOAddressSpace: mac: need base LimeAddressSpace: lime: need base WindowsHiberFileSpace32: No base Address Space WindowsCrashDumpSpace64BitMap: No base Address Space WindowsCrashDumpSpace64: No base Address Space HPAKAddressSpace: No base Address Space VMWareMetaAddressSpace: No base Address Space VirtualBoxCoreDumpElf64: No base Address Space VMWareAddressSpace: No base Address Space QemuCoreDumpElf: No base Address Space WindowsCrashDumpSpace32: No base Address Space Win10AMD64PagedMemory: No base Address Space WindowsAMD64PagedMemory: No base Address Space LinuxAMD64PagedMemory: No base Address Space AMD64PagedMemory: No base Address Space IA32PagedMemoryPae: No base Address Space IA32PagedMemory: No base Address Space OSXPmemELF: No base Address Space MachOAddressSpace: MachO Header signature invalid LimeAddressSpace: Invalid Lime header signature WindowsHiberFileSpace32: No xpress signature found WindowsCrashDumpSpace64BitMap: Header signature invalid WindowsCrashDumpSpace64: Header signature invalid HPAKAddressSpace: Invalid magic found VMWareMetaAddressSpace: VMware metadata file is not available VirtualBoxCoreDumpElf64: Unknown VBox core format version VMWareAddressSpace: Invalid VMware signature: 0x464c457f QemuCoreDumpElf: ELF error: did not find any PT_NOTE segment with CORE WindowsCrashDumpSpace32: Header signature invalid Win10AMD64PagedMemory: Incompatible profile WinXPSP2x86 selected WindowsAMD64PagedMemory: Incompatible profile WinXPSP2x86 selected LinuxAMD64PagedMemory: Incompatible profile WinXPSP2x86 selected AMD64PagedMemory: Incompatible profile WinXPSP2x86 selected IA32PagedMemoryPae: No valid DTB found IA32PagedMemory: No valid DTB found MachOAddressSpace: MachO Header signature invalid LimeAddressSpace: Invalid Lime header signature WindowsHiberFileSpace32: No xpress signature found WindowsCrashDumpSpace64BitMap: Header signature invalid WindowsCrashDumpSpace64: Header signature invalid HPAKAddressSpace: Invalid magic found VMWareMetaAddressSpace: VMware metadata file is not available VirtualBoxCoreDumpElf64: ELF Header signature invalid VMWareAddressSpace: Invalid VMware signature: 0xf000ff53 QemuCoreDumpElf: ELF Header signature invalid WindowsCrashDumpSpace32: Header signature invalid Win10AMD64PagedMemory: Incompatible profile WinXPSP2x86 selected WindowsAMD64PagedMemory: Incompatible profile WinXPSP2x86 selected LinuxAMD64PagedMemory: Incompatible profile WinXPSP2x86 selected AMD64PagedMemory: Incompatible profile WinXPSP2x86 selected IA32PagedMemoryPae: No valid DTB found IA32PagedMemory: No valid DTB found OSXPmemELF: ELF Header signature invalid FileAddressSpace: Must be first Address Space ArmAddressSpace: No valid DTB found

2016-08-30 12:21:09,282 [modules.processing.static] CRITICAL: You do not have the m2crypto library installed preventing certificate extraction: pip install m2crypto 2016-08-30 12:21:09,889 [modules.processing.network] ERROR: Error running httpreplay-based PCAP analysis Traceback (most recent call last): File "/home/alexander/cuckoo/modules/processing/network.py", line 764, in run results.update(Pcap2(pcap_path, self.get_tlsmaster()).run()) File "/home/alexander/cuckoo/modules/processing/network.py", line 695, in run r = httpreplay.reader.PcapReader(self.pcap_path) File "/usr/local/lib/python2.7/dist-packages/httpreplay/reader.py", line 25, in init self.pcap = dpkt.pcap.Reader(fp) File "/usr/local/lib/python2.7/dist-packages/dpkt/pcap.py", line 115, in init buf = self.f.read(FileHdr.__hdr_len) AttributeError: 'str' object has no attribute 'read' 2016-08-30 12:21:15,949 [lib.cuckoo.core.scheduler] INFO: Task #21: reports generation completed (path=/home/alexander/cuckoo/storage/analyses/21) 2016-08-30 12:21:16,039 [lib.cuckoo.core.scheduler] INFO: Task #21: analysis procedure completed

[doomedraven]

@fez219 i have solved that, change vm perfil in conf/memory.conf, you need specify perfil of vms.

@jbremer

        if self.voptions.basic.guest_profile:
            self.osprofile = self.voptions.basic.guest_profile
        else:
            self.osprofile = osprofile or self.get_osprofile()

This will trigger al the timeWinXPSP2x86 probably would be better remove that? Also looks like it ignores machinery config with profile value

[fez219]

@doomedraven Now I get a whole bunch of these messages:

2016-08-31 08:33:34,514 [volatility.debug] WARNING: NoneObject as string: No path, service isn't running 2016-08-31 08:33:34,517 [volatility.debug] WARNING: NoneObject as string: No path, service isn't running 2016-08-31 08:33:34,520 [volatility.debug] WARNING: NoneObject as string: No path, service isn't running 2016-08-31 08:35:00,190 [volatility.debug] WARNING: NoneObject as string: Pointer Local invalid 2016-08-31 08:35:00,194 [volatility.debug] WARNING: NoneObject as string: Pointer Local invalid

A bit of research on the issue says that these are on the Volatility side, and there's no fix yet. Is that the case? Either way, I am no longer getting a red error message from Volatility. That's a good step!

Also, I still need help on the m2crypto and httpreplay problem. Like I said, I have m2crypto and httpreplay installed, and am not sure why those functions still pull up errors.

[doomedraven]

Is not affect memory analysis for you so that is ok, vol is solved, about rest no ide ai dont have this problem, but from your output your system no detect them

[fez219]

@doomedraven yeah, I don't know what's up with that. But I checked, and Volatility is giving me useful, correct output. So thanks for that! Hopefully somebody else will comment and clear up the other two issues, but you just helped a lot! :)

[doomedraven]

@fez219 lets see

your error triggered here https://github.com/cuckoosandbox/cuckoo/blob/64c602ffc0fa7f271e5da55daabf850c391e250d/modules/processing/static.py#L285

so your problem is here

try:
    import M2Crypto
    HAVE_MCRYPTO = True
except ImportError:
    HAVE_MCRYPTO = False

can you open python console and type import M2Crypto? and post output? should be like in next image

[doomedraven]

my version M2Crypto==0.25.1

[acalarch]

Hello.

For the http relay issue. There appears to be an issue with the code in httprelay/reader.py It passes the path of the pcap to dpkt instead of a file object (@jbremer). It may be that dpkt once accepted a file path as an argument but it currently doesn't appear to.

To fix the issue on my cuckoo instance, i changed the code in reader.py to pass a fileobject to dpkt/pcap.py instead of a file path. You could also change the code in cuckoo/modules/processing/network.py to pass a file object instead of a path.. but i think httprelay/reader.py is meant to receive a filepath b/c the variable name is "fp".

ORIGINAL: (httprealy.reader.py)

    def __init__(self, fp):
        self.tcp = None
        self.udp = None
        self.values = []

        try:
            self.pcap = dpkt.pcap.Reader(fp)
        except ValueError as e:
            if e.message == "invalid tcpdump header":
                log.critical("Currently we don't support PCAP-NG files")
            self.pcap = None

MY PATCH VERSION:

 def __init__(self, fp):
        self.tcp = None
        self.udp = None
        self.values = []
        try:
        #PATCH
            pcap_file = open(fp)
            self.pcap = dpkt.pcap.Reader(pcap_file)
        except ValueError as e:
            if e.message == "invalid tcpdump header":
                log.critical("Currently we don't support PCAP-NG files")
            self.pcap = None`

[fez219]

@doomedraven Here's my m2crypto output:

import M2Crypto Traceback (most recent call last): File "", line 1, in File "/usr/local/lib/python2.7/dist-packages/M2Crypto/init.py", line 26, in from M2Crypto import (ASN1, AuthCookie, BIO, BN, DH, DSA, EVP, Engine, Err, File "/usr/local/lib/python2.7/dist-packages/M2Crypto/ASN1.py", line 15, in from M2Crypto import BIO, m2, util File "/usr/local/lib/python2.7/dist-packages/M2Crypto/BIO.py", line 10, in from M2Crypto import m2, util File "/usr/local/lib/python2.7/dist-packages/M2Crypto/m2.py", line 30, in from M2Crypto._m2crypto import * File "/usr/local/lib/python2.7/dist-packages/M2Crypto/_m2crypto.py", line 26, in __m2crypto = swig_import_helper() File "/usr/local/lib/python2.7/dist-packages/M2Crypto/_m2crypto.py", line 22, in swig_import_helper _mod = imp.load_module('m2crypto', fp, pathname, description) ImportError: /usr/local/lib/python2.7/dist-packages/M2Crypto/m2crypto.so: undefined symbol: SSLv2_method

@acalarch I'll try your fix and get back to you shortly!

[doomedraven]

@fez219 if @acalarch patch not working for you, which is your os?

[jbremer]

We should probably get rid of m2crypto altogether. It's not exactly the nicest library to work with..

[fez219]

@acalarch @doomedraven I just tried Acalarch's patch, no dice. I still get the same error. My host is Ubuntu 16.04, my guest Windows 7.

[jbremer]

@acalarch @fez219 Regarding the httpreplay bug, this was because you were using incompatible versions (see also requirements.txt for the correct version). However, I've recently bumped httpreplay to feature backwards compatibility with Cuckoo 2.0-rc1, so upgrading it should resolve that issue.

Regarding m2crypto I have yet to test.

[jbremer]

Btw, I unit tested the m2crypto stuff earlier this week; the following commit should contain all the information you need (that commit is for #863 though) https://github.com/cuckoosandbox/cuckoo/commit/328e9b694ea2442495379b9aadf3319661be3384. The volatility output is also related to Volatility itself, but we'll investigate that another time as in a normal setup it works'ish. Closing this issue as resolved for now, please reopen if required and let us know if you have any additional questions.

[keithjjones]

@jbremer I get the same issue today on Ubuntu that has the same output at @fez219. I'm using the latest dev version. Any ideas?

It seems to be a problem if you run virtualenv, from what I can tell. Running outside the virtualenv and apt-get install python-m2crypto will work but is not ideal.

[SparkyNZL]

OK So i got the same issue as well, and this is what i did to fix it

1. sudo pip uninstll M2Crypto
2. sudo apt-get install swig 3 sudo pip install M2Crypto

now i can import M2Crypto into python.

Cheers hope this helps.