Adding new tool request - the DumpsterDiver

[xep624]

Hi!

Did you think about enhancing the cs-suite and add new tools? I think it would be quite useful to add AWS MAcie-like tool the DumpsterDiver: https://github.com/securing/DumpsterDiver. I can add a feature to download the content from S3 bucket and then each file would be scanned via the DumpsterDiver in search of any hardcoded key, password or any pattern. Let me know what do you think about it.

Cheers! Pawel.

[shivankar-madaan]

Hi @xep624 I think this is really cool idea.Yea I think we can add this feature to download the buckets which are only public, instead downloading all of them(just an idea).I'm being bit worried about the size of the bucket contents as well, also should we download only text files?.

@jayeshchauhan any thoughts?

[xep624]

Hi @shivankar-madaan Why only public buckets? I rather thought that using cs-suite you can specify the keys to your bucket(s). The DumpsterDiver has an option of removing a file if there's no finding in it. 2 parallel process, when one is for downloading (for better performance it can support multithreading for parallel downloading) and the second one is for verifying a file. If nothing is found then the DumpsterDiver by default will be run with '-r' flag which will remove a file if nothing is found there. Regarding file types I think there should be only excluded picture/video files. If the tool cannot read a file (e.g. it's encrypted) then it cannot find anything there, so such file would be removed. What do you think about such idea?

[shivankar-madaan]

Hey @xep624 I was suggesting public buckets only as, only they pose a good amount of risk to be leaked.But very obvious, not a good practice either to store keys in private buckets as well. Regarding the files being kept or removed on the local system,Initially I was just assuming,that we show an alert on the html report the cs-suite generates(for the file having senstive info), with specifying the bucket and its respective path and I further assumed we would clear off all the files from the local system, after they being analysed (again I just assumed and just an idea). I agree that we should have video/images removed from the analysis part.

Just gaining more insights on DumpsterDiver, other than AWS keys, does it also detect API keys of other kinds.

Just for the heads up, cs-suite currently just requires read-only iam permissions.So basically we will have to add up one more permission of downloading S3 contents as well.

[xep624]

Hi @shivankar-madaan Sorry for late reply. Regarding removing all files and report findings in the html report - full agree, this is how should it work. Can DumpsterDiver detect API keys? yes it can it maybe customised only to look for API keys. Regarding permissions, having read-only permissions to the bucket should be enough and I don't think there is required special permission for downloading files. If you need any information, support - just please let me know!

[Electronickss]

other than AWS keys

This may have been added recently but

DumpsterDiver is a tool used to analyze big volumes of various file types in search of hardcoded secrets like keys (e.g. AWS Access Key, Azure Share Key or SSH keys) or passwords

Which, at the very least, would mean it would be interesting to use with Azure as well

[xep624]

Absolutely agree. It will just require other downloader - the rest would stay the same.

[shivankar-madaan]

yes I agree, we should definitely leverage this. I will get this added, else if anyone wants to work on this Pull Request are highly welcome :)