Output from IOCScanner is not tidy

[ogarrett]

The default output from IOCScanner should be easily machine-readable.

The current output is written to stdout, and resembles the following:

$ docker run -it --rm --name=deepfence-ioc-scanner -v $(pwd):/home/deepfence/output -v /var/run/docker.sock:/var/run/docker.sock deepfenceio/deepfence-ioc-scanner:latest -image-name deepfenceio/deepfence-ioc-scanner:latest
Initializing....
Scanning image deepfenceio/deepfence-ioc-scanner:latest for IOC...
INFO[2022-06-30 13:00:00] trying to connect to endpoint 'unix:///var/run/docker.sock' with timeout '10s'
INFO[2022-06-30 13:00:00] connected successfully using endpoint: unix:///var/run/docker.sock
INFO[2022-06-30 13:00:00] trying to connect to endpoint 'unix:///run/containerd/containerd.sock' with timeout '10s'
WARN[2022-06-30 13:00:10] could not connect to endpoint 'unix:///run/containerd/containerd.sock': context deadline exceeded
INFO[2022-06-30 13:00:10] trying to connect to endpoint 'unix:///run/k3s/containerd/containerd.sock' with timeout '10s'
WARN[2022-06-30 13:00:20] could not connect to endpoint 'unix:///run/k3s/containerd/containerd.sock': context deadline exceeded
INFO[2022-06-30 13:00:20] container runtime detected: docker
Scanning image /tmp/Deepfence/IOCScanning/df_deepfenceiodeepfenceiocscannerlatest/save-output.tar for IOCs...
include filescan.yar
include filescan.yar
include filescan.yar
include filescan.yar
include filescan.yar
include filescan.yar
include filescan.yar
[{description This rule is looking for B64 offsets of LazyNetToJscriptLoader which is a namespace specific to the internal version of the GadgetToJScript tooling.} {md5 7af24305a409a2b8f83ece27bb0f7900} {rev 4} {author FireEye}]
[{description Identifies GoRat malware in memory based on strings.} {md5 3b926b5762e13ceec7ac3a61e85c93bb} {rev 1} {author FireEye}]
[{date_created 2020-12-01} {date_modified 2020-12-01} {md5 d5d3d23c8573d999f1c48d3e211b1066} {rev 1} {author FireEye}]
[{date_created 2020-11-27} {date_modified 2020-11-27} {md5 d0a830403e56ebaa4bfbe87dbfdee44f} {rev 1} {author FireEye}]
[{author FireEye}]
[{author Akamai CSIRT} {description Rule to detect XOR DDos infection}]
    {
      "Image Layer ID": "76bf69f202ff803d5f037e1d2407965e6ed2e387ce887fcf05d7fbd1b415409a",
      "Matched Rule Name": "Hunting_GadgetToJScript_1",
      "String to Match": [   $s1 $s2 $s3],
      "Content to Match": ",,,GF6eU5ldFRvSnNjcmlwdExvYWRl,henlOZXRUb0pzY3JpcHRMb2Fk,YXp5TmV0VG9Kc2NyaXB0TG9hZGV",
      "Severity": "high",
      "Severity Score": 10.00,
      "Full File Name": "/tmp/Deepfence/IOCScanning/df_deepfenceiodeepfenceiocscannerlatest/ExtractedFiles/76bf69f202ff803d5f037e1d2407965e6ed2e387ce887fcf05d7fbd1b415409a/home/deepfence/usr/filescan.yar",
      "Match Rule Meta": [   value: This rule is looking for B64 offsets of LazyNetToJscriptLoader which is a namespace specific to the internal version of the GadgetToJScript tooling. value: 7af24305a409a2b8f83ece27bb0f7900 value: 4 value: FireEye],
    },
    {
      "Image Layer ID": "76bf69f202ff803d5f037e1d2407965e6ed2e387ce887fcf05d7fbd1b415409a",
      "Matched Rule Name": "APT_Backdoor_Win_GoRat_Memory",
      "String to Match": [                    $murica $murica $murica $murica $murica $murica $rat1 $rat1 $rat2 $rat3 $rat4 $rat5 $rat6 $rat7 $rat8 $rat9 $rat9 $rat10 $rat11 $winblows],
      "Content to Match": ",,,,,,,,,,,,,,,,,,,,murica,murica,murica,murica,murica,murica,rat/modules/socks.(*HTTPProxyClient).beacon,rat/modules/socks.(*HTTPProxyClient).beacon,rat.(*Core).generateBeacon,rat.gJitter,rat/comms.(*protectedChannel).SendCmdResponse,rat/modules/filemgmt.(*acquire).NewCommandExecution,rat/modules/latlisten.(*latlistensrv).handleCmd,rat/modules/netsweeper.(*netsweeperRunner).runSweep,rat/modules/netsweeper.(*Pinger).listen,rat/modules/socks.(*HTTPProxyClient).beacon,rat/modules/socks.(*HTTPProxyClient).beacon,rat/platforms/win/dyloader.(*memoryLoader).ExecutePluginFunction,rat/platforms/win/modules/namedpipe.(*dummy).Open,rat/platforms/win.(*winblows).GetStage",
      "Severity": "high",
      "Severity Score": 10.00,
      "Full File Name": "/tmp/Deepfence/IOCScanning/df_deepfenceiodeepfenceiocscannerlatest/ExtractedFiles/76bf69f202ff803d5f037e1d2407965e6ed2e387ce887fcf05d7fbd1b415409a/home/deepfence/usr/filescan.yar",
      "Match Rule Meta": [                    value: Identifies GoRat malware in memory based on strings. value: 3b926b5762e13ceec7ac3a61e85c93bb value: 1 value: FireEye],
    },
    {
      "Image Layer ID": "76bf69f202ff803d5f037e1d2407965e6ed2e387ce887fcf05d7fbd1b415409a",
      "Matched Rule Name": "Trojan_Macro_RESUMEPLEASE_1",
      "String to Match": [      $str00 $str01 $str02 $str03 $str04 $str05],
      "Content to Match": ",,,,,,For Binary As,Range.Text,Environ(,CByte(,.SpawnInstance_,.Create(",
      "Severity": "high",
      "Severity Score": 10.00,
      "Full File Name": "/tmp/Deepfence/IOCScanning/df_deepfenceiodeepfenceiocscannerlatest/ExtractedFiles/76bf69f202ff803d5f037e1d2407965e6ed2e387ce887fcf05d7fbd1b415409a/home/deepfence/usr/filescan.yar",
      "Match Rule Meta": [      value: 2020-12-01 value: 2020-12-01 value: d5d3d23c8573d999f1c48d3e211b1066 value: 1 value: FireEye],
    },
    {
      "Image Layer ID": "76bf69f202ff803d5f037e1d2407965e6ed2e387ce887fcf05d7fbd1b415409a",
      "Matched Rule Name": "APT_Builder_PY_REDFLARE_1",
      "String to Match": [       $1 $2 $3 $4 $5 $6 $7],
      "Content to Match": ",,,,,,,LOAD_OFFSET_32 = 0x612,LOAD_OFFSET_64 = 0x611,class RC4:,struct.pack('\u003cQ' if is64b else '\u003cL',stagerConfig['comms']['config'],_x86.dll,_x64.dll",
      "Severity": "high",
      "Severity Score": 10.00,
      "Full File Name": "/tmp/Deepfence/IOCScanning/df_deepfenceiodeepfenceiocscannerlatest/ExtractedFiles/76bf69f202ff803d5f037e1d2407965e6ed2e387ce887fcf05d7fbd1b415409a/home/deepfence/usr/filescan.yar",
      "Match Rule Meta": [       value: 2020-11-27 value: 2020-11-27 value: d0a830403e56ebaa4bfbe87dbfdee44f value: 1 value: FireEye],
    },
    {
      "Image Layer ID": "76bf69f202ff803d5f037e1d2407965e6ed2e387ce887fcf05d7fbd1b415409a",
      "Matched Rule Name": "APT_Backdoor_PS1_BASICPIPESHELL_1",
      "String to Match": [      $s1 $s2 $s3 $s4 $s5 $s6],
      "Content to Match": ",,,,,,function Invoke-Client(),function Invoke-Server,Read-Host 'Enter Command:',new-object System.IO.Pipes.NamedPipeClientStream(,new-object System.IO.Pipes.NamedPipeServerStream(, = iex $",
      "Severity": "high",
      "Severity Score": 10.00,
      "Full File Name": "/tmp/Deepfence/IOCScanning/df_deepfenceiodeepfenceiocscannerlatest/ExtractedFiles/76bf69f202ff803d5f037e1d2407965e6ed2e387ce887fcf05d7fbd1b415409a/home/deepfence/usr/filescan.yar",
      "Match Rule Meta": [      value: FireEye],
    },
    {
      "Image Layer ID": "76bf69f202ff803d5f037e1d2407965e6ed2e387ce887fcf05d7fbd1b415409a",
      "Matched Rule Name": "XOR_DDosv1",
      "String to Match": [       $st0 $st1 $st2 $st3 $st4 $st5 $st6],
      "Content to Match": ",,,,,,,BB2FA36AAA9541F0,md5=,denyip=,filename=,rmfile=,exec_packet,build_iphdr",
      "Severity": "high",
      "Severity Score": 10.00,
      "Full File Name": "/tmp/Deepfence/IOCScanning/df_deepfenceiodeepfenceiocscannerlatest/ExtractedFiles/76bf69f202ff803d5f037e1d2407965e6ed2e387ce887fcf05d7fbd1b415409a/home/deepfence/usr/filescan.yar",
      "Match Rule Meta": [       value: Akamai CSIRT value: Rule to detect XOR DDos infection],
    }include filescan.yar
include filescan.yar
{
  "Timestamp": "2022-06-30 13:00:31.400542592 +00:00",
  "Image Name": "deepfenceio/deepfence-ioc-scanner:latest",
  "Image ID": "6ad57ca08f568fc315a41354aeb18d3279e3165c7671134822c26ae088bedff4",
  "IOC": [

  ]
}

Recommendations

Scan results must go to STDOUT, and be a single JSON document Diagnostic output (INFO and the like) must go to STDERR All diagnostic messages should be prefixed by INFO/WARN etc and a timestamp

Default log level should be ERROR, so no messages are sent to STDERR on a successful scan

Rationale

This:

• follows standard unix practices for non-interactive scripts
• results in human-readable output (diagnostics and output are equally visible in the terminal)
• is easy to machine-parse (you can pipe to another tool e.g. jq and the output is not corrupted by diagnostics)

[ramanan-ravi]

@ogarrett I think it is ok to be verbose in stdout. Many tools are like that and they provide an option to save the output as a json/csv file which is machine readable.

 $ syft .
 ✔ Indexed .
 ✔ Cataloged packages      [1217 packages]
NAME                                               VERSION                             TYPE
@ampproject/remapping                              2.1.2                               npm
@babel/code-frame                                  7.12.11                             npm

syft --help

     --file string                    file to write the default report output to (default is STDOUT)
     --output stringArray             report output format, options=[json text table cyclonedx cyclonedx-json spdx-tag-value spdx-json] (default [table])

IOC Scanner also has this option https://github.com/deepfence/IOCScanner/issues/6

[ogarrett]

It's OK to be verbose when the tool is used interactively (at a terminal), but we should also support non-interactive use. Non-interactive use includes running in a pipe in a terminal, or running from a script.

For example, I might want to do something like:

IOCScanner | jq <jq expression> | <additional processing>

... either in a terminal or in a script.

The typical unix behaviour that enables this is:

• Output goes to stdout. If there's no output, write nothing to stdout (or perhaps an empty JSON document)
• All diagnostic messages go to stderr (no exceptions)
• Exit code is 0 or non-zero depending on a successful or unsuccessful operation (optional)

Note that syft does this correctly. The pretty animations and diagnostics messages are written to stderr, and the regular output goes to stdout. You can verify this by running syft . > /tmp/output.txt.

By all means, we can implement a --output option to configure where to write output to in addition to defaulting to stdout for output.

[ogarrett]

BTW - diagnostic messages don't have to be prefixed by <SEVERITY> [<TIMESTAMP>]. This is common for persistent daemon services where the events are processed by (for example) syslog, but is not common practice for single-operation tools.

Let's be consistent - either use them everywhere, or nowhere.

[ogarrett]

Update, after further investigation:

The YaRadare binary, using logrus, functions correctly with respect to logging, and writes logs to STDERR. However, when running in docker with a pseudo-tty, it merges STDOUT and STDERR into a single output stream (STDOUT), so logs and output are conflated.

The solution is to run the docker container without allocating a tty, as:

$ docker run -i --rm --name=deepfence-yaradare     -v /var/run/docker.sock:/var/run/docker.sock     deepfenceio/deepfence-yaradare:latest --image-name node:latest > /tmp/docker-no-t.txt
time="2022-07-08 13:16:23" level=info msg="trying to connect to endpoint 'unix:///var/run/docker.sock' with timeout '10s'"
time="2022-07-08 13:16:23" level=info msg="connected successfully using endpoint: unix:///var/run/docker.sock"
time="2022-07-08 13:16:23" level=info msg="trying to connect to endpoint 'unix:///run/containerd/containerd.sock' with timeout '10s'"
time="2022-07-08 13:16:33" level=warning msg="could not connect to endpoint 'unix:///run/containerd/containerd.sock': context deadline exceeded"
time="2022-07-08 13:16:33" level=info msg="trying to connect to endpoint 'unix:///run/k3s/containerd/containerd.sock' with timeout '10s'"
time="2022-07-08 13:16:43" level=warning msg="could not connect to endpoint 'unix:///run/k3s/containerd/containerd.sock': context deadline exceeded"
time="2022-07-08 13:16:43" level=info msg="container runtime detected: docker\n"

logrus uses a different output format in this case, but it's still usable.

The output is still untidy and not well-formatted JSON. Now that we've determined the STDOUT/STDERR problem and workaround/solution, I'll raise a separate issue concerning the output.