search

airbnb / binaryalert

BinaryAlert: Serverless, Real-time & Retroactive Malware Detection.
https://binaryalert.io
Apache License 2.0
1.41k stars 187 forks source link

Failed Unit tests #108

Closed adutta14 closed 6 years ago

adutta14 commented 6 years ago

Hello,

Environment Details:-

OS : Ubuntu 16.04 Python version : 3.5

I followed the steps given in the documentation and i get the below errors when i try to run the unit test cases:-

$ ./manage.py unit_test E....EEEE......EEEEEEEEFEEEEEEEE.EE...........E.EE.E.EEEE.E.........

ERROR: tests.lambda_functions.analyzer.analyzer_aws_lib_test (unittest.loader._FailedTest)

ImportError: Failed to import test module: tests.lambda_functions.analyzer.analyzer_aws_lib_test Traceback (most recent call last): File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path module = self._get_module_from_name(name) File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name import(name) File "/home/ubuntu/binaryalert/tests/lambda_functions/analyzer/analyzer_aws_lib_test.py", line 6, in from lambda_functions.analyzer import analyzer_aws_lib, binary_info, yara_analyzer File "/home/ubuntu/binaryalert/lambda_functions/analyzer/analyzer_aws_lib.py", line 193 previous_s3_objects: Set[str] = set() ^ SyntaxError: invalid syntax

====================================================================== ERROR: test_analyze_lambda_handler (tests.lambda_functions.analyzer.main_test.MainTest) Verify return value, logging, and boto3 calls when multiple files match YARA rules.

Traceback (most recent call last): File "/home/ubuntu/binaryalert/tests/lambda_functions/analyzer/main_test.py", line 99, in setUp from lambda_functions.analyzer import main File "/home/ubuntu/binaryalert/lambda_functions/analyzer/main.py", line 15, in from lambda_functions.analyzer import analyzer_aws_lib, binary_info, yara_analyzer File "/home/ubuntu/binaryalert/lambda_functions/analyzer/analyzer_aws_lib.py", line 193 previous_s3_objects: Set[str] = set() ^ SyntaxError: invalid syntax

====================================================================== ERROR: test_analyze (tests.lambda_functions.analyzer.yara_analyzer_test.YaraAnalyzerTest) Analyze returns the expected list of rule matches.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1576, in _inner return f(*args, *kw) File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(args, **keywargs) File "/home/ubuntu/binaryalert/tests/lambda_functions/analyzer/yara_analyzer_test.py", line 60, in test_analyze mock_subprocess.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_analyze_match_with_target_path (tests.lambda_functions.analyzer.yara_analyzer_test.YaraAnalyzerTest) Match additional rules if the target path is provided.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1576, in _inner return f(*args, *kw) File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(args, **keywargs) File "/home/ubuntu/binaryalert/tests/lambda_functions/analyzer/yara_analyzer_test.py", line 84, in test_analyze_match_with_target_path mock_subprocess.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_analyze_no_matches (tests.lambda_functions.analyzer.yara_analyzer_test.YaraAnalyzerTest) Analyze returns empty list if no matches.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1576, in _inner return f(*args, *kw) File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(args, **keywargs) File "/home/ubuntu/binaryalert/tests/lambda_functions/analyzer/yara_analyzer_test.py", line 76, in test_analyze_no_matches mock_subprocess.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_batcher_empty_bucket (tests.lambda_functions.batcher.main_test.MainTest) Batcher does nothing for an empty bucket.

Traceback (most recent call last): File "/home/ubuntu/binaryalert/tests/lambda_functions/batcher/main_test.py", line 26, in setUp from lambda_functions.batcher import main File "/home/ubuntu/binaryalert/lambda_functions/batcher/main.py", line 35 self._keys: List[str] = [] ^ SyntaxError: invalid syntax

====================================================================== ERROR: test_batcher_invoke_with_continuation (tests.lambda_functions.batcher.main_test.MainTest) Invoke the batcher with a continuation token.

Traceback (most recent call last): File "/home/ubuntu/binaryalert/tests/lambda_functions/batcher/main_test.py", line 26, in setUp from lambda_functions.batcher import main File "/home/ubuntu/binaryalert/lambda_functions/batcher/main.py", line 35 self._keys: List[str] = [] ^ SyntaxError: invalid syntax

====================================================================== ERROR: test_batcher_multiple_messages (tests.lambda_functions.batcher.main_test.MainTest) Batcher enqueues 2 SQS messages.

Traceback (most recent call last): File "/home/ubuntu/binaryalert/tests/lambda_functions/batcher/main_test.py", line 26, in setUp from lambda_functions.batcher import main File "/home/ubuntu/binaryalert/lambda_functions/batcher/main.py", line 35 self._keys: List[str] = [] ^ SyntaxError: invalid syntax

====================================================================== ERROR: test_batcher_one_full_batch (tests.lambda_functions.batcher.main_test.MainTest) Batcher enqueues the configured maximum number of objects in a single SQS message.

Traceback (most recent call last): File "/home/ubuntu/binaryalert/tests/lambda_functions/batcher/main_test.py", line 26, in setUp from lambda_functions.batcher import main File "/home/ubuntu/binaryalert/lambda_functions/batcher/main.py", line 35 self._keys: List[str] = [] ^ SyntaxError: invalid syntax

====================================================================== ERROR: test_batcher_one_object (tests.lambda_functions.batcher.main_test.MainTest) Batcher enqueues a single S3 object.

Traceback (most recent call last): File "/home/ubuntu/binaryalert/tests/lambda_functions/batcher/main_test.py", line 26, in setUp from lambda_functions.batcher import main File "/home/ubuntu/binaryalert/lambda_functions/batcher/main.py", line 35 self._keys: List[str] = [] ^ SyntaxError: invalid syntax

====================================================================== ERROR: test_batcher_re_invoke (tests.lambda_functions.batcher.main_test.MainTest) If the batcher runs out of time, it has to re-invoke itself.

Traceback (most recent call last): File "/home/ubuntu/binaryalert/tests/lambda_functions/batcher/main_test.py", line 26, in setUp from lambda_functions.batcher import main File "/home/ubuntu/binaryalert/lambda_functions/batcher/main.py", line 35 self._keys: List[str] = [] ^ SyntaxError: invalid syntax

====================================================================== ERROR: test_batcher_sqs_errors (tests.lambda_functions.batcher.main_test.MainTest) Verify SQS errors are logged and reported to CloudWatch.

Traceback (most recent call last): File "/home/ubuntu/binaryalert/tests/lambda_functions/batcher/main_test.py", line 26, in setUp from lambda_functions.batcher import main File "/home/ubuntu/binaryalert/lambda_functions/batcher/main.py", line 35 self._keys: List[str] = [] ^ SyntaxError: invalid syntax

====================================================================== ERROR: test_build_all (tests.lambda_functions.build_test.BuildTest) Verify that the top-level build function executes each individual builder.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/lambda_functions/build_test.py", line 119, in test_build_all build_analyzer.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_build_batcher (tests.lambda_functions.build_test.BuildTest) Verify that a valid zipfile is generated for the batcher Lambda function.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/lambda_functions/build_test.py", line 88, in test_build_batcher mock_print.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_build_dispatcher (tests.lambda_functions.build_test.BuildTest) Verify that a valid zipfile is generated for the dispatcher Lambda function.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/lambda_functions/build_test.py", line 96, in test_build_dispatcher mock_print.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_build_downloader (tests.lambda_functions.build_test.BuildTest) Verify list of bundled files for the downloader.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/lambda_functions/build_test.py", line 107, in test_build_downloader mock_pip.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_dispatcher_invalid_message (tests.lambda_functions.dispatcher.main_test.MainTest) Dispatcher discards invalid SQS messages.

Traceback (most recent call last): File "/home/ubuntu/binaryalert/tests/lambda_functions/dispatcher/main_test.py", line 26, in setUp from lambda_functions.dispatcher import main File "/home/ubuntu/binaryalert/lambda_functions/dispatcher/main.py", line 45 payload: Dict[str, List[str]] = {'Records': [], 'SQSReceipts': []} ^ SyntaxError: invalid syntax

====================================================================== ERROR: test_dispatcher_invokes_analyzer (tests.lambda_functions.dispatcher.main_test.MainTest) Dispatcher flattens multiple messages and invokes an analyzer.

Traceback (most recent call last): File "/home/ubuntu/binaryalert/tests/lambda_functions/dispatcher/main_test.py", line 26, in setUp from lambda_functions.dispatcher import main File "/home/ubuntu/binaryalert/lambda_functions/dispatcher/main.py", line 45 payload: Dict[str, List[str]] = {'Records': [], 'SQSReceipts': []} ^ SyntaxError: invalid syntax

====================================================================== ERROR: test_dispatcher_no_messages (tests.lambda_functions.dispatcher.main_test.MainTest) Dispatcher doesn't do anything if there are no SQS messages.

Traceback (most recent call last): File "/home/ubuntu/binaryalert/tests/lambda_functions/dispatcher/main_test.py", line 26, in setUp from lambda_functions.dispatcher import main File "/home/ubuntu/binaryalert/lambda_functions/dispatcher/main.py", line 45 payload: Dict[str, List[str]] = {'Records': [], 'SQSReceipts': []} ^ SyntaxError: invalid syntax

====================================================================== ERROR: test_copy_all_binaries (tests.lambda_functions.downloader.copy_all_test.CopyAllTest) Test the top-level copy function with real multiprocessing.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/lambda_functions/downloader/copy_all_test.py", line 79, in test_copy_all_binaries from lambda_functions.downloader import copy_all File "/home/ubuntu/binaryalert/lambda_functions/downloader/copy_all.py", line 76 process_name: str = self.name # type: ignore ^ SyntaxError: invalid syntax

====================================================================== ERROR: test_copy_with_errors (tests.lambda_functions.downloader.copy_all_test.CopyAllTest) Test top-level copy function with injected errors.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/lambda_functions/downloader/copy_all_test.py", line 102, in test_copy_with_errors from lambda_functions.downloader import copy_all File "/home/ubuntu/binaryalert/lambda_functions/downloader/copy_all.py", line 76 process_name: str = self.name # type: ignore ^ SyntaxError: invalid syntax

====================================================================== ERROR: test_configure_with_defaults (tests.manage_test.BinaryAlertConfigTestFakeFilesystem) Test configure() when all variables have already had set values.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/manage_test.py", line 144, in test_configure_with_defaults mock_encrypt.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_configure_with_no_defaults (tests.manage_test.BinaryAlertConfigTestFakeFilesystem) Test configure() without any values set - no defaults should print.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/manage_test.py", line 170, in test_configure_with_no_defaults mock_encrypt.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_encrypt_cb_api_token (tests.manage_test.BinaryAlertConfigTestRealFilesystem) Verify that token encryption is done correctly.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/manage_test.py", line 246, in test_encrypt_cb_api_token mock_getpass.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_apply (tests.manage_test.ManagerTest) Validate order of Terraform operations.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/manage_test.py", line 309, in test_apply mock_print.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_build (tests.manage_test.ManagerTest) Calls lambda_build function (tested elsewhere).

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/manage_test.py", line 315, in test_build mock_build.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_clone_rules (tests.manage_test.ManagerTest) Calls clone_rules_from_github (tested elsewhere).

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/manage_test.py", line 328, in test_clone_rules mock_clone.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_compile_rules (tests.manage_test.ManagerTest) Calls compile_rules (tested elsewhere).

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/manage_test.py", line 335, in test_compile_rules mock_compile.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_configure (tests.manage_test.ManagerTest) Calls BinaryAlertConfig:configure() (tested elsewhere).

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/manage_test.py", line 343, in test_configure mock_configure.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_deploy (tests.manage_test.ManagerTest) Deploy docstring includes each executed command and runs each.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/manage_test.py", line 357, in test_deploy mock_test.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_destroy (tests.manage_test.ManagerTest) Destroy asks whether S3 objects should also be deleted.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/manage_test.py", line 370, in test_destroy mock_input.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== ERROR: test_live_test (tests.manage_test.ManagerTest) Live test wrapper raises TestFailureError if appropriate.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/manage_test.py", line 380, in test_live_test mock_live_test.assert_called_once() File "/usr/lib/python3.5/unittest/mock.py", line 583, in getattr raise AttributeError(name) AttributeError: assert_called_once

====================================================================== FAIL: test_build_analyzer (tests.lambda_functions.build_test.BuildTest) Verify that a valid zipfile is generated for analyzer Lambda function.

Traceback (most recent call last): File "/usr/lib/python3.5/unittest/mock.py", line 1157, in patched return func(*args, **keywargs) File "/home/ubuntu/binaryalert/tests/lambda_functions/build_test.py", line 77, in test_build_analyzer 'YEXTEND_LICENSE' File "/home/ubuntu/binaryalert/tests/lambda_functions/build_test.py", line 39, in _verify_filenames self.assertEqual(expected_filenames, filenames) AssertionError: Items in the second set but not the first: './'

Ran 68 tests in 3.166s

FAILED (failures=1, errors=32) TEST FAILED: Unit tests failed

Thanks!!

austinbyers commented 6 years ago

Only Python 3.6+ is supported because BinaryAlert takes advantage of type annotations. I'll make this a little clearer in the documentation, but to install Python 3.6 in ubuntu:

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.6 python3.6-dev python3-pip
sudo -H pip3 install --upgrade pip
python3.6 --version

virtualenv -p python3.6 venv
source venv/bin/activate
pip3 install -r requirements.txt
adutta14 commented 6 years ago

Alright. Let me try that and get back to you. Appreciate your response.

adutta14 commented 6 years ago

Hello Austin,

Apologies for responding late. Now 1 of the unit test failed.

:~/binaryalert$ ./manage.py deploy ................................E.........................................

ERROR: test_build_downloader (tests.lambda_functions.build_test.BuildTest) Verify list of bundled files for the downloader.

Traceback (most recent call last): File "/usr/lib/python3.6/unittest/mock.py", line 1171, in patched arg = patching.enter() File "/usr/lib/python3.6/unittest/mock.py", line 1243, in enter original, local = self.get_original() File "/usr/lib/python3.6/unittest/mock.py", line 1217, in get_original "%s does not have the attribute %r" % (target, name) AttributeError: <module 'pip' from '/home/ubuntu/binaryalert/venv/lib/python3.6/site-packages/pip/init.py'> does not have the attrbute 'main'

Ran 74 tests in 3.021s

FAILED (errors=1) TEST FAILED: Unit tests failed

Am i missing anything here?

Thanks, Abhishek Dutta

austinbyers commented 6 years ago

Ah good catch. It turns out that the latest version of pip no longer supports being imported from a Python script. You can either fast-forward to the latest BinaryAlert commit (this was fixed in #115), or you can stick with the latest stable version of BinaryAlert:

git clone --branch v1.1.0 https://github.com/airbnb/binaryalert
adutta14 commented 6 years ago

Hello Austin,

I believe you are referring to this commit " 08043c010fcc73c0b7c08ce6a9eaf2c01b92d5dc". Just making sure it is the right one.

Also, i just want to make sure that once i fast forward to the commit, will my current AWS resources be impacted in any ways as i have few important files stored in the s3 bucket and i don't want to loose them.

Thanks, Abhishek Dutta

austinbyers commented 6 years ago

Yes, that looks like the right commit. Whenever you run ./manage.py apply, terraform will always preview its execution to give you a chance to confirm it. Your existing resources should not be affected, but if by some crazy chance they are, terraform will tell you what it's about to do and you'll have a chance to cancel.

Terraform also won't destroy an S3 bucket until all the objects have been cleared, unless you've enabled force_destroy in the configuration options.

adutta14 commented 6 years ago

Thanks a lot Austin. It worked and i am able to successfully run my tests for the binaries. One last question. How can i make sure that whatever custom rules i add in the future, i can have a proof that the files are tested against the new rules as well and not only the default rules. So, which logs can be referred for that?.

Currently, the cloudwatch logs only tells whether the file matched any YARA rule or not.

austinbyers commented 6 years ago

You're right - at the moment, BinaryAlert only logs positive YARA matches, otherwise it would be very noisy (#104 is an open issue to add alerting on negative matches).

Any .yara or .yar file in the rules/ directory will be compiled into BinaryAlert. You can run ./manage.py compile_rules to generate the compiled YARA file which BinaryAlert uses. Then you can open a Python terminal to explore it:

import yara
rules = yara.load('compiled_yara_rules.bin')
for r in rules:
    print(r.identifier)

To look at the list of YARA rules that are actually running in production:

  1. Goto the BinaryAlert analyzer Lambda function via AWS Console (console.aws.amazon.com) => Lambda => binaryalert_analyzer.
  2. Under "Actions", there is an option to "export function." This lets you download the Lambda function as a .zip file, so you can inspect it.
  3. Once you download and extract the zipfile, there is a compiled_yara_rules.bin file, which you can explore as above.

Alternatively, BinaryAlert exports a custom CloudWatch metric called YaraRules. The metric is visible on the autogenerated CloudWatch dashboard or you can view the metric directly. This tells you how many YARA rules BinaryAlert found in production (you can make sure the number is higher than the default)

adutta14 commented 6 years ago

Awesome. So, i used the Cloudwatch option and checked the metrics in there. The count value shows 3161. Does, it mean that it analyze through these many rules in PROD in total for all the binaries?

Metric name: BinaryAlert YaraRules Region: us-west-2 Period: 1 Hour Statistic: Maximum Unit: Count Value: 3,161 Time: 2018-05-21 18:00

austinbyers commented 6 years ago

Right - every file is currently being scanned against 3,191 YARA rules

adutta14 commented 6 years ago

Alright. Answers all my questions. Thank you for all your help.