Refactor, SentinelOne support, and miscellaneous improvements

[jholtmann]

This is a fairly major rewrite of the surveyor tool. My initial goal was to implement SentinelOne support but I realized that doing so would be very difficult given the project's architecture. The problems were as follows:

• Implementing a new product required adding checks to various methods in common.py as well as changes to surveyor.py.
• Authentication for each product was handled in common.py instead of being delegated to each individual product.
• process_search and nested_process_search both required results to be returned to surveyor.py. This does not work well for SentinelOne due to the 1 request per minute rate limit for Deep Visibility searches via the API. The correct way to handle this is to combine various process searches into a single query, but this was not possible given the old architecture.

I addressed these issues in the following ways:

1. Refactor for CBR/CBTH/Defender products:

• Refactored common.py into an abstract class that defines a product interface.
• Refactored all supported products to inherit from the new abstract base class.
• Refactored surveyor.py to take advantage of the new interface.
• Implemented load.py which can dynamically load any product class based on its unique name.

The benefits of this refactor are as follows:

• Authentication logic for each product is now encapsulated in the product itself. Before, authentication had to be handled in surveyor.py with branching paths depending on how the authentication for the particular product functioned (profile vs credential file).
• Implementing a new product is now as simple as creating a new class in products/ and making sure it has a unique product attribute. load.py will automatically expose the new product to surveyor.py.

2. Refactored process_search and nested_process_search:

• Both methods no longer return any data. Instead, the results are stored in a class attribute and can be retrieved via product.get_results(). This returns a dictionary where the keys represent tags used to identify the results and the values represent the results themselves.
• This change maintains the ability for surveyor.py to display per-process/query result totals while also granting new products greater flexibility to change the way they handle search operations internally.

3. Miscellaneous changes made:

• Simplified the code flow for certain operations in surveyor.py (e.g. having three branched open() calls when the only change between them was the file name).
• Enabled multiple search flags to be used in tandem (e.g. you can specify both --query and --deffile).
• Implemented logging via python's logging module.
• Logs are written to the logs folder. This can be configured via the CLI parameter --log-dir.
• Added type hints. Note that this does reduce the versions of Python the project is compatible with. The benefit is that it becomes easier to ensure errors are not introduced during development.
• Refactored code to comply with PEP8.
• Minor changes to improve documentation, logging, and error handling.
• Deffiles that reside in the definitions folder can now be resolved by the --deffiles CLI option without needing to specify the definitions folder or the .json extension.
• Created a requirements.txt file to list required dependencies for development environments.

Resolved issues:

• Closes #31: Implemented progress bar for deffiles parsing and added --no-progress CLI option to suppress the progress bar.
• Closes #2: Added --no-file CLI option to print results to STDOUT instead of a CSV file. Implicitly sets --no-progress to True.
• Closes #52: Implemented SentinelOne product.

Implemented support for SentinelOne:

• SentinelOne support is implemented using the SentinelOne API and Deep Visibility.
• Searches are combined to reduce the number of API requests made to SentinelOne.
• The 1 request per minute rate limit for Deep Visibility queries is observed.
• Added s1 click command.

Potential Breaking Changes:

• Click was modified to leverage commands. This means that existing CLI strings will no longer function as the product no longer needs to be prepended with --.
• Verified that the changes have not impacted the cbr product implementation's survey results.
• I do not have access to cbth or defender product instances so those have not been tested.

New module dependencies:

• Added requests to the list of required module installs in setup.py. Note that this module was already required for the defender product.
• tqdm to resolve issue #31.

To-Do:

• These changes need to be tested on all implemented products.
• Documentation needs to be updated.
• There are likely some common CBR search terms that have not been mapped to their S1 equivalents.
• S1 query merging logic can be improved.
• Add timestamp to output for all products.

[jholtmann]

Summary of the most recent changes:

• Simplified naming of products:
  • response -> cbr
  • threathunter -> cbc
  • defender/atp -> dfe
  • All currently implemented products now include an additional output column containing a timestamp. This closes #45.
  • A version check in surveyor.py will now print an error message if the user launches surveyor with a Python version <3.9.
  • Resolved a bug in the dfe product that resulted in a 400 error for all queries.
  • dfe product now raises an error if the user's credential file is missing
  • I have successfully performed queries on all implemented products.
  • Quoted process names in the cbc engine when the process name contains a space. Before this change I was receiving an API error when querying for process names with spaces in the name. Might be related to #3, though that issue relates to the CBR product.

[jholtmann]

Proposed changes to the wiki. As wiki pages can't be included in a pull request I can add these once this PR is merged. Please let me know if there are any other wiki pages that need to be updated @rc-abodkins.

Changelog

March 2022

• Support for SentinelOne
• Refactor of the code base to make adding support for new products easier
• Implemented logging
• Resolved various outstanding issues
• See PR #53 for more details

Getting Started

Prerequisites

You need the following to use Surveyor:

• Python 3.9+. You can download the latest version of Python here.

Save your SentinelOne API Credentials

To use Surveyor with SentinelOne, you need the following information from SentinelOne:

• API Key for user with permission to execute Deep Visibility queries
• URL of your SentinelOne server

Save your configuration information in an INI file as follows:

[profile_name_here]
url=<URL>
token=<API KEY>

Alternatively, you can omit the token field and specify your API token via the environment variable S1_TOKEN.

When you run Surveyor, specify the path of the INI file with the --creds option (e.g. py surveyor.py s1 --creds s1_creds.ini). Note that unless otherwise specified with --profile, Surveyor uses the credentials and URL provided by the [default] header.

Other Changes

It might make sense to create per-product wiki pages that document any additional options implemented for those products. For example, the SentinelOne page could document the --site-id, --account-id, and --account-name options.

[rc-abodkins]

@jholtmann to the Getting Started wiki we need to add that Python 3.9 is required for Surveyor to work as well as the new commands and their structure.

I think a per product Wiki is the right way to go and then we can also move all the credential pieces to that and additional information.

[jholtmann]

@rc-abodkins I have updated my comment above to reflect that addition. The README in this pull request also mentions the 3.9+ requirement.

[EricMich]

unsubscribe

On Wed, Mar 30, 2022 at 12:37 PM Jonathan Holtmann @.***> wrote:

@rc-abodkins https://github.com/rc-abodkins I have updated my comment above to reflect that addition. The README in this pull request also mentions the 3.9+ requirement.

— Reply to this email directly, view it on GitHub https://github.com/redcanaryco/surveyor/pull/53#issuecomment-1083548708, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADGKEWCSMBRN7ZQNMVJBXHDVCSUQ3ANCNFSM5QHJ2VKQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>