Option to Raise FILESYSTEM and WEBSCREENSHOT with base64 blobs

[nicpenning]

Draft code things

[nicpenning]

FYI - Needs work and is not currently functioning or well thought out.

[TheTechromancer]

It's a good start. Tomorrow I'll see if I can pitch in and get it working.

[TheTechromancer]

@nicpenning are you able to allow edits from maintainers?

[nicpenning]

Looks like I already had that set.

[TheTechromancer]

Okay, I made a few changes:

• Added global file_blobs and folder_blobs options
• Wrote tests
• Added feature for both WEBSCREENSHOT and FILESYSTEM

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 98.66667% with 1 line in your changes missing coverage. Please review.

Project coverage is 92%. Comparing base (eeae1cb) to head (e8a45e0).

Files	Patch %	Lines
bbot/core/event/base.py	98%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## stable #1438 +/- ## ====================================== - Coverage 92% 92% -0% ====================================== Files 321 321 Lines 20685 20749 +64 ====================================== + Hits 19014 19057 +43 - Misses 1671 1692 +21 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[nicpenning]

Phenomenal!

[nicpenning]

Please let me know how I can test with custom/flag options, and I can run do some trial runs to see how it shakes out.

[TheTechromancer]

Please let me know how I can test

@nicpenning you can test like this:

bbot -t evilcorp.com -c file_blobs=true folder_blobs=true

This will automatically attach binary blobs to any WEBSCREENSHOT and FILESYSTEM events. Do be aware that if the FILESYSTEM event is a folder (i.e. a git repo), the blob contains a compressed tar.gz archive.

[TheTechromancer]

@nicpenning have you got a chance to test?

[nicpenning]

Not yet, but I will test as soon as I can! My initial test didn't work but I might not have had my config options or code updated properly.

[nicpenning]

So far works great! Only downside is that the blob event that gets emitted to the console takes up a lot of lines (like 600K and in some cases I imagine it could be millions). It is bearable, but makes it more difficult to watch the scans progression using the default output.

Going to test the strict options to ensure I only get blobs for the appropriate scope.

[nicpenning]

So does FILESYSTEM == filedownload?

I get all of the screenshots as PNGs in the output.ndjson but never the filedownload but shouldn't I need to set that configuration for the filedownload to use the blob feature? It doesn't seem the pdf, docx, etc.. that are in the filedownload directory are in the event at all (not just missing the blob).

Running this command bbot -t evilcorp.com -c file_blobs=true folder_blobs=true output_modules.http.siem_friendly=true modules.filedownload.base_64_encoded_file=true -m filedownload httpx gowitness

{"type": "WEBSCREENSHOT", "id": "WEBSCREENSHOT:4ec4b0a8e3ce89c7a85d9bb9b81d91dd652cccea", "data": {"filename": "/home/yams/.bbot/scans/heightened_jeremy/gowitness/screenshots/https-evilcorp.com.png", "url": "https://evilcorp.com/inside-evilcorp", "blob": "iVBORw0KG..."}

TL:DR - Screenshots seem to work, but the FILESYSTEM / filedownload module does not emit an event (so blob definitely does not exist).

[nicpenning]

Here is a real world example: bbot -t "https://eicar.org/download-anti-malware-testfile/" -c file_blobs=true folder_blobs=true output_modules.http.siem_friendly=true modules.filedownload.base_64_encoded_file=true -m filedownload httpx gowitness

Files downloaded, but none are in the events (output.ndjson).

Otherwise, everything else is working as expected.

[TheTechromancer]

Hmm I think this branch needs merged with the latest dev, so it includes this PR:

[TheTechromancer]

I'm going to go ahead and merge this. We can make tweaks in a followup PR if needed.

[nicpenning]

Sounds great