Find Replacement for Caldera

[4ndr3w6]

• Operating System Version: Mac Mojave (10.14.4)
• Provider (VirtualBox/VMWare): VMWare
• Vagrant Version: 2.2.4
• Packer Version: 1.2.5
• Are you using stock boxes (downloaded) or were they built from scratch using Packer? Stock
• Is the issue reproducible or intermittent? Reproducible

Please verify that you are building from an updated Master branch before filing an issue.

Using master :)

Description of the issue:

Hey @clong! Thanks so much for your hard work into this project. Huge fan of Detection Lab!! I'm running into an issue with Caldera. Using a fresh clone from "master" and the latest stock boxes, the Logger is failing to come "up" due to Caldera. I have also tried removing Caldera prior to build by altering the Vagrantfile, but I am still having no luck in bringing the logger "up." This may be due to MITRE releasing Caldera 2.0 this morning at BSidesCharm?

Thank you so much again!!

Link to Gist Containing Build Logs:

For some reason vagrant is not making logs for me. Here is the output from terminal:

logger: [+] applied 1 packs logger: [+] applied 32 queries logger: [+] applied 1 packs logger: Added monitor of '/home/vagrant/kolide-quickstart/osquery_result'. logger: Added monitor of '/home/vagrant/kolide-quickstart/osquery_status'. logger: Installing Caldera... logger: Cloning into 'caldera'... logger: /tmp/vagrant-shell: line 223: cd: /home/vagrant/caldera/caldera: No such file or directory The SSH command responded with a non-zero exit status. Vagrant assumes that this means the command failed. The output for this command should be in the log above. Please read the output to determine what went wrong.

[clong]

Oh, interesting - that may have done it! I'm going to try rebuilding now and we'll see how it goes.

[clong]

Yep, I'm able to repro this. Looking into it.

[4ndr3w6]

Thanks; and thanks again for your work/time on this! Really appreciate this contribution to the community, especially given you put a lot of your own time into making this project a success. :)

[clong]

I’ll have a fix/update for this out by EOD

[clong]

I'm pretty disappointed with Caldera 2.0. Although installation is easier and has less dependencies the interface is pretty awful and unintuitive. The provided "agent" is literally just a powershell script that pipes output to stdout and from my limited experimentation, has pretty limited capabilities and may be more focused for Mac/Linux systems. I think this is actually a pretty significant regression from the old version and I'm thinking it might be better to just remove it from the project entirely and look for a suitable replacement.

Many of the old capabilities (enumerating hosts on the network, running mimikatz, etc) are all missing from the new version and it's way less manageable to install from a scripting standpoint.

[clong]

I'm going to remove support for Caldera as they don't have releases I can pin to and I don't think version 2.0 is even worth including in the lab. I'll look for alternatives and am definitely open to suggestions (e.g. Atomic Red Team, etc).

[JonZeolla]

+1 ART

[4ndr3w6]

@clong Thanks for troubleshooting and remediating this issue so quickly. Especially on a Saturday night!! I just did an update to my master branch and had no problems bringing the logger "up." Sorry you had to remove Caldera. The only other open source project as an alternative to ART that I am aware of is Endgame's RTA (https://www.endgame.com/blog/technical-blog/introducing-endgame-red-team-automation). However, I have not used it yet. Cheers!

[clong]

ART has been added in https://github.com/clong/DetectionLab/pull/266

RTA appears to not have been updated in at least 9 months - I don't think it's worth adding.

I'm going to leave this with ART for now, but feel free to add suggestions to this thread.

[daniel-infosec]

@clong Thoughts on https://github.com/praetorian-code/purple-team-attack-automation (disclaimer: I'm one of the primary devs)? We're working independently to build our own fork of DetectionLab that integrates that project. And I'd be happy to chat about seeing if it can work for you as a fully integrated part of DetectionLab.

[aapplebaum]

@clong I'm one of the leads for CALDERA -- sorry to hear that the 2.0 release didn't match your expectations and that you'll be removing it. This thread is helpful for us, but we'd definitely appreciate any other feedback you might have; feel free to shoot us an email at caldera@mitre.org.

One thing I'd note is that the v1.0 capabilities are still in 2.0; they're now just part of the "adversary mode" plugin (https://github.com/mitre/adversary). By default CALDERA does not start with adversary mode enabled due to its dependence on Mongo, but so long as Mongo is running you can enable the adversary/classic mode plugin. There's more details on plugin/architecture on the main CALDERA README.

[clong]

Hey @aapplebaum - thanks for the info!

I hope I didn't come off too harsh, I was just dismayed by the experience of configuring operations in Caldera 2.0 and it's good to know that functionality still exists! Although I understand the extensibility benefits are huge, the tool feels less approachable now. Is the Crater RAT still functional in 2.0? The Powershell agent (although neat that it's only memory resident) isn't really suitable to "install" in an environment and keep active and persistent.

I'm definitely open to feedback if I've overlooked things here!

[aapplebaum]

@clong -- I totally understand, and definitely not too harsh! We pushed this update without a ton of documentation behind it (I have a to-do to write a blog post, though we have a talk here that goes through some of the changes) and I can appreciate the amount of work you'd put into getting CALDERA into your pipeline in the past.

CALDERA now has two disjoint modes, implemented as plugins. Chain mode is the new 2.0 capability -- this is designed to run/orchestrate atomic unit tests, where the operator more explicitly configures CALDERA on what it should do and when, stringing together unit tests stored in YML files. Adversary mode is the classic CALDERA capability, albeit in a new GUI (but with a lot of back-end enhancements).

Regarding agents -- the PowerShell agent is only used by Chain mode; Adversary mode still uses Crater and CAgent. The adversary mode agents are now stored with the adversary mode plugin here so you no longer need to pull them from the crater repo. Adversary mode also streamlines the installation somewhat; it ships by default with a PowerShell script to install CAgent now.

(I'd note you could save Chain mode's PowerShell agent as a script and make it persistent; you'd still have to make explicit adversary profiles for Chain mode for this to be useful for your use case, so adversary mode might be the better fit)

Hope this helps, and happy to answer any additional questions -- we also have a slack I can add you to if you shoot us an email.

[clong]

Closing this out as Atomic Red Team was added and I don't have plans to add any other automation/testing frameworks at the moment