[SALTSTACK] [CLI] Limited support for Ubuntu Server (gap between complete and packages only installation)

securyeti commented 6 years ago

Problem Statement

There is limited support for SIFT on Ubuntu server builds. Using sift-cli complete mode fails entirely and packages-only results in a significant number of tools failing to fully install.

Desired Outcome

Support for Ubuntu Server builds added, with full installation of all packages/tools that are not dependant on a GUI environment.

Background

There is a gap between the complete installation and the packages-only version of the SIFT install, which impacts functionality on Ubuntu Server builds.

Based on prior issues / comments, packages-only appears to be recommended for Ubuntu server builds as the complete install assumes there is a desktop environment. However, packages-only excludes a significant number of useful tools from being incorporated into the final build (i.e. pdf-tools, regripper, etc.), for immediate use by an end user.

This seems to stem from https://github.com/sans-dfir/sift-saltstack/blob/master/sift/pkgs.sls, which invokes only three of the salt subtrees (sift.repos, sift.packages, sift.python-packages).

Many of the missing tools appear to be present in https://github.com/sans-dfir/sift-saltstack/tree/master/sift/scripts (needs further review on the other subtrees).

Related Issues:

291
264 (may be a separate issue)
275

Solution Options

A few possible options for solving this:

Add an opinionated server option (everything not dependant on a desktop environment) to sift-cli
Update packages-only to install the additional tools that are not dependant on a GUI
Add a sift-cli option to specify which of the subtrees to pull in (this seems like the most onerous change)
Add an option in sift-cli to pull from a fork of the sift-saltstack codebase
Update docs to reflect current state and provide further detail on server builds

In terms of pros/cons:

Option 1/2 are probably the most user friendly, with option 1 providing better backwards compatibility for anyone currently using packages-only
Option 3 seems like a pretty significant overhead, and is probably less manageable long term

Next steps

If there's interest in having one of these options implemented (or something else that achieves the objective) I'm happy to do so and submit pull requests for the changes, but wanted to get the team's view before going off in a direction that you may not be interested in. I'd personally favour option 1, but open to discussion.

If there's interest in proceeding, it may require some of the .sls files to be moved to a specific "desktop-only" subtree (possibly one for desktop-only, one for server-only). Any thoughts?

ekristen commented 6 years ago

Thanks for the amazing write up. I do apologize for the delay in response. I agree there is a gap. I will take a look.

mgg1776 commented 5 years ago

@securyeti I have interest in your option 1 or 2, or really any option that installs all tools and configs not dependent upon a GUI.

FWIW, up until recently when I'd install on Ubuntu Server 16.04 with --packages-only, I'd only get errors on GUI-dependent components, but the command line tools I use most (e.g., Volatility, log2timeline/plaso, and bulk_extractor) would function fine.

In the past couple of weeks I've been trying to do a fresh installation, I've been having a lot or problems, such command line tools that used to work apparently not being installed anymore. Specifically, Volatiltiy, log2timeline, and bulk_extractor appear not to be installed with --packages-only on Ubuntu Server 16.04.

So, it seems at one point --packages-only worked to some degree on Server installations, but something changed that broke what compatibility there was.

I can provide log files/error messages if interested, but didn't want to hijack this thread.

securyeti commented 5 years ago

Thanks @mgg1776, appreciate the feedback! No need for log files / error messages (at least at this point), but definitely good background to have.

@ekristen, I'm going to take a look at implementing option 1 or 2, any further thoughts from you would be appreciated.

securyeti commented 5 years ago

I've created pull requests (for both sift-cli and for sift-saltstack) to implement option one described in the original post.

Changes for sift-cli are:

New mode "server" added

Changes for sift-saltstack are:

new state file for the server build server.sls
moving 3 of the state files into a new gui module (with the vm.sls state file updated to utilise this to ensure no changes)

I've tested building sift-cli (with a couple of additional changes not in the pull request to point it at the fork of sift-saltstack) and using it to deploy the updated sift-saltstack code, on both ubuntu server 16.04 and ubuntu desktop 16.04. This deploys without error.

I've also tested a sift install (using release 1.7.1) on ubuntu desktop 16.04, followed by an upgrade with the newer sift-cli and sift-saltstack versions in the pull requests. This also works (although I did hit an error that was outlined in a previous issue), the fix for which is documented below in case anyone hits the same issue in future.

Error message: 'module' object has no attribute 'SSL_ST_INIT'
Fix: sudo pip install -U pyopenssl

@ekristen, @mgg1776, let me know if you have any feedback.

ekristen commented 5 years ago

@securyeti thanks for the work! I'll take a look and let you know.

The CLI changes seem pretty straight forward.

ekristen commented 5 years ago

I'm going to close this for now. I have your PRs open still. I will work these changes in soon. I appreciate your patience.

teamdfir / sift