Uncompress from YaRa dispatch randomly failing

[aleksf]

Hi, I am new to Stoq. Only started playing with it yesterday. Stoq installed from Git. Python 3.6.9.

I am using command below to process Maildir files with further dispatch to YaRa. I am testing with single RFC822-compliant forwarded email sample that contains ZIP attachment. ZIP contains PE. What I am trying to achieve is trivial:

• YaRa identifies attachment as ZIP and dispatches to decompress, which extracts and archives PE

stoq run --log-level debug -P dirmon -R yara -A filedir -C filedir -C stdout  -s smtp \
  --plugin-opts yara:dispatch_rules=`pwd`/dispatcher.yar \
  yara:worker_rules=`pwd`/stoq.yar  \
  dirmon:source_dir=`pwd`/samples \
  filedir:archive_dir=`pwd`/archive \
  filedir:results_dir=`pwd`/results \
  smtp:always_dispatch=hash,mimetype smtp:archive_attachments=True smtp:extract_iocs=True

Command produces inconsistent results. In most cases it fails with error:

    "errors": [
        {
            "error": "worker:failed to scan: File \"/home/rtops/.stoq/plugins/decompress/decompress.py\", line 151, in scan ; KeyError: 'mimetype'",
            "plugin_name": "decompress",
            "payload_id": "6c1c06c3-19c6-423d-b6ef-73e4af995b2c"
        }
    ],

On every run mimetype is identified properly by plugin and in payload meta: payload meta:

            "payload_meta": {
                "should_archive": true,
                "should_scan": true,
                "extra_data": {
                    "charset": null,
                    "content-description": null,
                    "disposition": "attachment",
                    "filename": "INVOICE COPY CONFIRMATION.pdf.zip",
                    "type": "application/zip"

mimetype plugin:

            "workers": {
                "mimetype": {
                    "mimetype": "application/zip"
                },

Occasionally it works and I get PE extracted and archived. Please help to identify the issue.

[aleksf]

Please disregard, apparently was related to testing with Mutt MUA. Extraction with YaRa dispatcher works with following config:

[core]
log_level: DEBUG
providers: dirmon
archivers: filedir
connectors: filedir,stdout
dispatchers: yara
[dirmon]
source_dir=/home/stoq/Maildir/new
[filedir]
archive_dir=/home/stoq/analysis/archive
results_dir=/home/stoq/analysis/results
[smtp]
always_dispatch=hash,exif,mimetype,vtmis-search,tika
archive_attachments=True
extract_iocs=True
omit_body=True
[yara]
dispatch_rules=/home/stoq/stoqyara/dispatcher.yar
worker_rules=/home/stoq/stoqyara/rules/index-custom.yar
[vtmis-search]
apikey=