Add CarbonBlack downloader

austinbyers commented 7 years ago

Overview

Size: Extra Large

CarbonBlack automatically uploads new binaries that it finds on endpoints; users who have CarbonBlack can now optionally enable a CarbonBlack downloader Lambda function to copy binaries from CarbonBlack into BinaryAlert.

The downloader can be enabled by running the new python3 manage.py configure command to allow the user to set the CarbonBlack URL and encrypt its API key.

Additionally, python3 manage.py cb_copy_all allows users to copy the entire CarbonBlack binary corpus into BinaryAlert in one go.

Full documentation will be added in the next PR.

Change Summary

Adds lambda_functions/downloader
- Added a enable_carbon_black_downloader Terraform variable. All downloader resources are created only if the downloader is explicitly enabled.
Changes to manage.py:
- Adds configure and cb_copy_all commands
- Renames test command to unit_test (to distinguish it from live_test)
- Refactored configuration management into a separate class
- Adds unit test coverage for the first time
Pylint has been slightly relaxed to allow longer variable names and more branches
Travis email notifications disabled
Updates pip requirements to their latest versions
Includes type annotations in all new code

Resolves: #29 (add downloader) Resolves: #48 (additional name prefix validation) Contributes to: #34 (type annotations)

Tested

CI

Added unit tests for downloader code as well as most of manage.py. As you can see from the commit history, mocking everything correctly was a huge pain. In the future, I think we should remove moto entirely (we already have to do our own Dynamo and S3 mocks due to 2 separate moto issues)

Test Deploy: Downloader Disabled (Default)

$ python3 manage.py deploy
ERROR: name_prefix "" does not match format [a-z][a-z0-9_]{3,50}
Please run "python3 manage.py configure"

$ python3 manage.py configure
AWS Region (us-east-1):
Unique name prefix, e.g. "company_team": ba_test_638
Enable the CarbonBlack downloader [yes/no]? (no): no
Updated configuration successfully saved to terraform/terraform.tfvars!

$ python3 manage.py deploy
...
Apply complete! Resources: 41 added, 0 changed, 0 destroyed.

$ python3 manage.py live_test
Live test succeeded!

$ python3 manage.py cb_copy_all
ERROR: CarbonBlack downloader is not enabled.
Please run "python3 manage.py configure"

Test Deploy: Enable Downloader

After the previous deploy, we can easily re-configure and re-deploy:

$ python3 manage.py configure
AWS Region (us-east-1):
Unique name prefix, e.g. "company_team" (ba_test_638):
Enable the CarbonBlack downloader [yes/no]? (no): yes
CarbonBlack URL: [URL redacted]
CarbonBlack API token (only needs binary read access):
Terraforming KMS key...
aws_kms_key.carbon_black_credentials: Creation complete
aws_kms_alias.encrypt_credentials_alias: Creation complete
Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
Encrypting API token...
Updated configuration successfully saved to terraform/terraform.tfvars!

$ python3 manage.py deploy
Apply complete! Resources: 8 added

$ python3 manage.py cb_copy_all
[Sampled Output]
2017-08-28 18:49:39,630 INFO   Start 32 consumers
2017-08-28 18:49:40,042 DEBUG  Enqueuing CopyTask [#0] MD5: ...
2017-08-28 18:49:40,045 DEBUG  Enqueuing CopyTask [#1] MD5: ...
2017-08-28 18:49:40,046 DEBUG  Enqueuing CopyTask [#2] MD5: ...
2017-08-28 18:49:40,046 INFO   [Consumer-1] Executing CopyTask [#0] MD5: ...
2017-08-28 18:49:40,047 INFO   [Consumer-2] Executing CopyTask [#1] MD5: ...
2017-08-28 18:49:40,047 INFO   [Consumer-3] Executing CopyTask [#2] MD5: ...
... More logs ...
2017-08-28 18:49:55,089 INFO   [Consumer-1] Exiting
2017-08-28 18:49:57,675 INFO   [Consumer-2] Exiting
2017-08-28 18:50:07,184 INFO   [Consumer-3] Exiting
2017-08-28 18:50:07,184 INFO   All CopyTasks Finished!

Reviewers

to: @ryandeivert cc: @mime-frame @airbnb/binaryalert-maintainers

coveralls commented 7 years ago

Coverage increased (+3.6%) to 90.134% when pulling bb60c79c8ab768311ea9a795dcc3e85ad6c55cd6 on abb--downloader into 6134b33c9e65c37f3523508b03123858fb9e1676 on master.

coveralls commented 7 years ago

Coverage increased (+3.6%) to 90.139% when pulling b4b8ae46e9ce140578f86787eea7698173e74b43 on abb--downloader into 548fbfb9fd913a74381ff58afed46a60f36b5312 on master.

coveralls commented 7 years ago

Coverage increased (+3.6%) to 90.139% when pulling be709300a670f4d94c5a54a3fafdf1c2693b04f0 on abb--downloader into 548fbfb9fd913a74381ff58afed46a60f36b5312 on master.

austinbyers commented 7 years ago

@ryandeivert PTAL

I've squashed the commits to simplify things and addressed all of the feedback. Be sure to look at the downloader code again because your last review was an an older commit for some reason

ryandeivert commented 7 years ago

@austinbyers your'e right - I had been stepping through commits since this PR was so large :)

austinbyers commented 7 years ago

@ryandeivert Thanks for your review! I should have done a better job of keeping track of and squashing the commits. I did another deploy just to test everything one last time and once the Travis tests pass I'll go ahead and merge

coveralls commented 7 years ago

Coverage increased (+3.6%) to 90.139% when pulling bb0e217eccc61d2a1518a09a74638b0ec32403e8 on abb--downloader into 548fbfb9fd913a74381ff58afed46a60f36b5312 on master.

airbnb / binaryalert