GCP Install Issues/502 error

[bryanCoteChang]

Hi,

I attempted to install Security Monkey on both CentOS and Ubuntu without success. I have received the following error during the gclient sync step . (Below is the error from my CentOS machine, but they are the same disregarding the different directories.):

.
.
.
________ running '/usr/bin/python sdk/build/vs_toolchain.py update' in '/home/bryancote/dart-sdk'

________ running '/usr/bin/python sdk/tools/generate_buildfiles.py' in '/home/bryancote/dart-sdk'
Command failed: /home/bryancote/dart-sdk/sdk/buildtools/linux64/gn gen --check out/DebugIA32 --args=is_release=false is_clang=false dart_zlib_path="//runtime/bin/zlib" use_goma=false dart_use_fallback_root_certificates=true target_cpu="x86" goma_dir="None" dart_use_tcmalloc=true dart_target_arch="ia32" is_asan=false is_product=false is_debug=true dart_runtime_mode="develop" dart_host_pub_exe="" dart_debug=true is_msan=false host_cpu="x86" is_tsan=false target_os="linux"
output: ERROR at //build/prebuilt_dart_sdk.gni:12:28: Script returned non-zero exit code.
_prebuilt_dart_exe_trial = exec_script("gn_run_binary.py",
                           ^----------
Current dir: /home/bryancote/dart-sdk/sdk/out/DebugIA32/
Command: python -- /home/bryancote/dart-sdk/sdk/build/gn_run_binary.py exec_script /home/bryancote/dart-sdk/sdk/tools/sdks/linux/dart-sdk/bin/dart --version
Returned 1.
stderr:

Traceback (most recent call last):
  File "/home/bryancote/dart-sdk/sdk/build/gn_run_binary.py", line 61, in <module>
    sys.exit(main(sys.argv))
  File "/home/bryancote/dart-sdk/sdk/build/gn_run_binary.py", line 54, in main
    result = run_command(args)
  File "/home/bryancote/dart-sdk/sdk/build/gn_run_binary.py", line 24, in run_command
    subprocess.check_output(command, stderr=subprocess.STDOUT)
  File "/usr/lib64/python2.7/subprocess.py", line 568, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

See //runtime/vm/BUILD.gn:6:1: whence it was imported.
import("../../build/prebuilt_dart_sdk.gni")
^-----------------------------------------
See //BUILD.gn:44:5: which caused the file to be included.
    "runtime/vm:patched_sdk",
    ^-----------------------
Error: Command '/usr/bin/python sdk/tools/generate_buildfiles.py' returned non-zero exit status 1 in /home/bryancote/dart-sdk
Hook '/usr/bin/python sdk/tools/generate_buildfiles.py' took 10.71 secs

[bryanCoteChang]

I followed the posted instructions and finally got it stood up. While running setup.py I was getting a bunch of errors about module's not having attributes. I eventually found this issue and modified my installed versions with respect to the pip freeze shown in one of the comments. Unfortunately, the application has entered a fail loop and I'm getting 502 errors and I'm auto-logged in as anonymous. Any help on this would be awesome. Also, is there a docker container somewhere that I could use in production instead?

[bryanCoteChang]

This is what's going on in supervisor:

[bryanCoteChang]

[bryanCoteChang]

Good news, it's finally installed. Ran this to get setup.py to complete: sudo pip install --upgrade distribute Ran this to get rid of the 502 error: sudo mkdir /home/www-data sudo chown -R www-data:www-data www-data sudo usermod -d /home/www-data -m www-data sudo chown -R www-data:www-data /var/log/security_monkey/* Bad news, it's not loading anything: Nothing in the error logs anywhere. I noticed this in the config-deploy.py, I'm assuming I need to add stuff there: I'm hoping there's something in here that might help: https://github.com/Netflix/security_monkey/pull/655

[scriptsrc]

New docs do instruct the user to run sudo pip install --upgrade setuptools which is very similar to your --upgrade distribute command.

[scriptsrc]

Thanks @bryanCoteChang .

I'm going to close this ticket, but if I run into problems with #655 I'll review the permissions you provided.

[bryanCoteChang]

@monkeysecurity I apologize for the potentially stupid questions, but I can't seem to add a GCP account for Security Monkey to monitor: It defaults to AWS. I've tried using the CLI without success: Help seems to throw an error as well: I incrementally installed each dependency in my virtual environment. Now it can't find my SECURITY_MONKEY_SETTINGS environment variable - even after I export it. (Triple checked for spelling/directory errors.)

Going to try scrap this vm and start over following the Quickstart directions verbatim. In the meantime, would it be possible to see an example of the add_account_gcp command?

[scriptsrc]

You said:

I incrementally installed each dependency in my virtual environment.

That seems odd to me. You should just need to do this:

cd /usr/local/src
sudo git clone --depth 1 --branch master https://github.com/Netflix/security_monkey.git
cd security_monkey
sudo virtualenv venv
sudo pip install --upgrade setuptools
sudo python setup.py install

add_account_aws and add_account_gcp

$ python manage.py add_account_aws
usage: manage.py add_account_aws [-h] -n NAME [--thirdparty] [--active]
                             [--notes NOTES] --id IDENTIFIER
                             [--update-existing]
                             [--canonical_id CANONICAL_ID]
                             [--s3_name S3_NAME] [--role_name ROLE_NAME]

$ python manage.py add_account_gcp
usage: manage.py add_account_gcp [-h] -n NAME [--thirdparty] [--active]
                             [--notes NOTES] --id IDENTIFIER
                             [--update-existing] [--creds_file CREDS_FILE]

The error you see is the system cannot find flask-script.

> pip freeze | grep script
flask-script==0.6.3

[scriptsrc]

I'll bring up a GCP instance and run through the quickstart too.

[scriptsrc]

FYI - I do recommend cloning from the develop branch instead of the master branch.

[scriptsrc]

Example:

$ python manage.py add_account_gcp -n my-project --active --id my-project

[bryanCoteChang]

Hi Patrick!

Thank you for your help and running through the Quickstart!! Great news, with your assistance, it's up and running!

I still had to do the permissions thing, but nbd.

I ran the add_account_gcp command, using my project's name, but I'm not getting any data. Do i need to add any additional files/configurations so it will start pulling data from GCP? Thank you again for your help and please see the log below for the error:

req = service.list(**kwargs)
  File "build/bdist.linux-x86_64/egg/googleapiclient/discovery.py", line 739, in method
    (name, pvalue, regex))
TypeError: Parameter "project" value "SecurityMonkey" does not match the pattern "(?:(?:[-a-z0-9]{1,63}\.)*(?:[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?):)?(?:[0-9]{1,19}|(?:[a-z0-9](?:[-a-z0-9]{0,61}[a-z0-9])?))"
2017-04-11 23:13:47,107 ERROR: Job "run_change_reporter (trigger: interval[1:00:00], next run at: 2017-04-12 00:13:46.679327)" raised an exception [in build/bdist.linux-x86_64/egg/apscheduler/scheduler.py:520]
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/apscheduler/scheduler.py", line 512, in _run_job
    retval = job.func(*job.args, **job.kwargs)
  File "/usr/local/src/security_monkey/security_monkey/scheduler.py", line 32, in run_change_reporter
    reporter.run(account, interval)
  File "/usr/local/src/security_monkey/security_monkey/reporter.py", line 56, in run
    (items, exception_map) = monitor.watcher.slurp()
  File "/usr/local/src/security_monkey/security_monkey/watchers/gcp/gce/firewall.py", line 71, in slurp
    return slurp_items()
  File "/usr/local/lib/python2.7/dist-packages/cloudaux-1.1.8-py2.7.egg/cloudaux/gcp/decorators.py", line 132, in decorated_function
    itm, exc = func(*args, **kwargs)
  File "/usr/local/src/security_monkey/security_monkey/watchers/gcp/gce/firewall.py", line 57, in slurp_items
    rules = list_firewall_rules(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/cloudaux-1.1.8-py2.7.egg/cloudaux/gcp/decorators.py", line 41, in decorated_function
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/cloudaux-1.1.8-py2.7.egg/cloudaux/gcp/gce/firewall.py", line 17, in list_firewall_rules
    **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/cloudaux-1.1.8-py2.7.egg/cloudaux/gcp/utils.py", line 85, in gce_list
    resp = req.execute()
  File "build/bdist.linux-x86_64/egg/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "build/bdist.linux-x86_64/egg/googleapiclient/http.py", line 840, in execute
    raise HttpError(resp, content, uri=self.uri)
HttpError: <HttpError 403 when requesting https://www.googleapis.com/compute/v1/projects/XXXXXXX/global/firewalls?alt=json returned "Required 'compute.firewalls.list' permission for 'projects/XXXXXXX'">

The link's output is as follows:

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "required",
    "message": "Login Required",
    "locationType": "header",
    "location": "Authorization"
   }
  ],
  "code": 401,
  "message": "Login Required"
 }
}

[scriptsrc]

It looks like it is complaining that your service account does not have the compute.firewalls.list permission.

Does your service account have the Project Viewer and the Security Reviewer roles?

[scriptsrc]

I'm now seeing a similar (but slightly different) GCP permissions issue. Looking for a solution

[bryanCoteChang]

Sorry for the inconveniences! Cannot thank you enough!

[antoine-castex]

Same problem as Bryan , not getting any data from GCP

where to find the logs ?

when i have created the gcp account i don't have give creds_file, what is the formal of this ? no information about this param in the guide

monkey add_account_gcp