possible to output JSON, instead of plaintext?

[poona]

No description provided.

Should be possible. So all the API log lines in a list, with each API log line as one item in the list? Or is it each API name as a key, and its arguments as params list? Can you paste a sample here to understand how you want it?

[evandrix]

For example, sample output from invoking APIMiner.exe --app C:\Windows\System32\notepad.exe: <synchronisation>-<0,0x00000000> NtCreateMutant([mutant_handle]0x000000C8, [desired_access]2031617, [initial_owner]0, [mutant_name]"Local\SM0:4976:168:WilStaging_02")

_Complete file below for reference: apiminer_traces.4640718.pid_4976.txt_

becomes:

{
    "kind":"synchronisation",
    "api":"NtCreateMutant",
    "params": {
        "mutant_handle":0x000000c8,
        "desired_access":2031617,
        "initial_owner":0,
        "mutant_name":"Local\SM0:4976:168:WilStaging_02"
    }
}

Repeat this process of string => JSON for every line in the log file, i.e. JSON Lines format.

My intended use case, is to use your tool to dynamically detonate malware samples (in batch mode), within a controlled, instrumented sandbox environment, then automatically extract IoCs from the execution trace output (like a mutex in this case) for further processing.

[evandrix]

I notice the use of BSON in your code. Is it the same as the BSON found in MongoDB? If so, then producing output in BSON format is also acceptable, perhaps for efficiency's sake, then subsequently using something like bsondump to dump it out to JSON, or just mongoimporting the BSON output directly into MongoDB would work as well.

[xrkk]

This is a standalone version of Cuckoo Monitor, and cuckoo monitor logs api calls in bson format, so this shall not be a problem at all.

[poona]

@evandrix If you want pure json format, the best option is to use Vanilla version of regular cuckoo setup, since you want to automate detonate and extract of samples in batch mode and the best way to do that is using a separate VM that comes with a full fledged cuckoo setup. APIMiner is a modified standalone command line version of Cuckoo Monitor, so API logs wise it should remain the same as Cuckoo, but it has been modified to suit the use-case meant for single malware executions in your existing Malware Anaysis Setup/VM, so that you don't have to re-open/re-spawn the whole Cuckoo setup/VM.

[evandrix]

yeah sure, a full-fledged cuckoo setup is certainly an option, but given that the latest version v2.0.7 is still on Python2, some of my clients may require it to be migrated to Python3, since Python2 EOL'ed already, which is why I was planning to strip it down myself i.e. DIY the management of the Sandbox VM + throw APIMiner into the VM as my "agent.py" in Cuckoo terms, and then stream out the logs from the VM to elsewhere.

care to briefly comment on the main enhancements APIMiner has over Cuckoo Monitor?

[poona]

@evandrix

In that case it does make sense.

The most important enhancements,

1. Make it standalone. This is what drove the introduction of APIMiner. The current cuckoo monitor can't be run as a standalone binary. This is a huge drawback imo, because as analysts all I want is a monitoring agent that I can use to detonate the file in my EXISTING analysis VM. I don't want to bring up a whole VM infra to get logs for a malware.
2. Change the injection mechanism from needing an outside external agent. Cuckoo relies on an external injector exectuable + python agent that listens for commands to inject the monitor into a new process. Definitely didn't want any extra agent, python or otherwies which APIMiner has to depend on.
3. Change the logging mechanism to log to a file on disk, instead of current pipe based streaming option.
4. Streamlined and simplified the config file. Just one config option in the config file now.

Future immediate enhancements already in development,

1. Completely remove the need for config file, so that any config option can be supplied via the command line. This greatly simplifies setup and shipping of APIMiner, and also execution steps needed to run any malware.
2. Provide a new Process Stop Functionality, that should again greatly help in analyzing non memory resident malwares.

[evandrix]

1. Make it standalone. This is what drove the introduction of APIMiner. The current cuckoo monitor can't be run as a standalone binary. This is a huge drawback imo, because as analysts all I want is a monitoring agent that I can use to detonate the file in my EXISTING analysis VM. I don't want to bring up a whole VM infra to get logs for a malware.
2. Change the injection mechanism from needing an outside external agent. Cuckoo relies on an external injector exectuable + python agent that listens for commands to inject the monitor into a new process. Definitely didn't want any extra agent, python or otherwies which APIMiner has to depend on.

!! EXACTLY !!

re:3. Actually, I prefer it to be streamed + piped out to elsewhere. re:4. yeah, the config now needs to live at precisely C:\apiminer_config.txt, but I can live with that.

[poona]

re.3 Our use case for APIMiner was mainly for malware analysts and reverse engineering, since most automation cases are covered with a full vm infra like Cuckoo. And as analysts simplicity was really important and nothing comes as neat as a text file(even json is great). If we allowed it to be piped out, then we need a secondary listener/process for the incoming logs.