Suricata extracted file info currently broken

[jgajek]

There is an issue with the current 3.2beta1 release of Suricata that causes the file id field to be inconsistently logged in eve.json and files-json.log. Since Cuckoo's Suricata module relies on this field to construct a list of extracted files, the analysis report and the "Suricata Files" page in the Django UI will contain only a subset of the files that were actually extracted by Suricata and present within the files.zip archive.

The issue appears to be related to the order in which Suricata runs the file-store and file-info output modules, and was addressed in a subsequent commit (https://github.com/inliniac/suricata/commit/f4b165de945beaa9b03981c0b84880845ac587c3) which is not yet in the packaged 3.2beta version.

I verified that Suricata as compiled from latest sources logs the file_id for all extracted files correctly. Moreover, it also logs the HTTP content type, which would be useful to include in the Cuckoo analysis report.

Is anyone seeing this issue in the 2.x Suricata release?

[doomedraven]

for me suricata 3.1.x stable works fine, 2.x worked fine too

[jgajek]

Hmm... these two commits to the Suricata module by @seanthegeek suggest that the incorrect "stored" flag and missing "id" have been a thing for quite a while:

https://github.com/spender-sandbox/cuckoo-modified/commit/1bb34ec0df3db16543b37b886672c1d44dfc5b69#diff-b3d3634e9c950fcd01c81c2fb0b7d197

https://github.com/spender-sandbox/cuckoo-modified/commit/d193639295ac45366071fc2a0b0072a5f4d8ea3c#diff-b3d3634e9c950fcd01c81c2fb0b7d197

Here is an example, with multiple runs against the same captured PCAP would result in Suricata file counts of anywhere between 41 and 59:

The actual number of files that Suricata extracted from this PCAP is 118:

This issue would have been somewhat masked by the fact that the Suricata file count in Cuckoo's expanded dashboard was broken until recently, and would always be 0.