CrowdStrike / Cloud-AWS

A collection of projects supporting AWS Integration
MIT License
146 stars 82 forks source link

Manual installation is not working #237

Closed schikkam closed 8 months ago

schikkam commented 1 year ago

AWS Security integration was successful:

Installation link used : https://github.com/CrowdStrike/Cloud-AWS/tree/main/Security-Hub/install

Here is the flow I tried:

Screenshot from 2023-06-07 21-56-11-InstallationSnapshot

Once after installation is completed , I tried do start the service but it didn't start. I'm unsure of logs to look into as I don't see anything in /var/logs/...

Tried to execute the python script by looking at service file but that too failed with some HTTP error.

Screenshot from 2023-06-07 21-56-50-Status_And_Manul_PyExecution

Please let me know if any steps missed in docs?.

jshcodes commented 1 year ago

Hi @schikkam -

The second screenshot looks like a permission error.

Do you know which user account you're wanting to run under? (Looks like you setup a fig user as well.) We should install the python requirements as that user, and then confirm they have access to the contents of /usr/shar/fig.

Side note: Logs are also in this same folder. You'll have one named after your stream (starts with a period) and one for the service in general, fig-service.log.

schikkam commented 1 year ago

Yes @jshcodes , I've installed it as fig user only. Let me check logs and get back to you if anything on it. Installation and packages were installed properly as I didn't see any error while installation. How does it talk to Falcon on cloud without giving API details? Is there a configuration missing prior starting main.py?

jshcodes commented 1 year ago

The two environment variables on the lambda, CLIENT_ID_PARAM and CLIENT_SECRET_PARAM hold the name of the two AWS Parameter Store parameters that store your API client ID and secret. Whatever values you specified for these two environment variables, we need to create parameters that match and insert your ID / secret where appropriate.

This particular piece is poorly documented for the manual process, we'll get this updated.

schikkam commented 1 year ago

Okay. So here is what I understood. Please correct me I'm misunderstood.

  1. Create Two Parameters in AWS Parameter Store CLIENT_ID_PARAM , CLIENT_SECRET_PARAM and point these to my falcon account client/secret.
  2. Make sure the node must have access to AWS parameter though IAM role or through creds.
  3. Start service now and see.

Thanks for your quick response. I will let you know the status!

jshcodes commented 1 year ago

Close!

  1. Create two environment variables on the lambda function, called CLIENT_ID_PARAM (which holds the name of the parameter that holds your client ID in Parameter Store) and CLIENT_SECRET_PARAM (which holds the name of the parameter that holds your client secret in Parameter Store).
  2. Create the two matching parameters within AWS Parameter Store, and populate them with the appropriate value from your API key.
  3. Start the service.
schikkam commented 1 year ago

I see a zip fie in this location :https://github.com/CrowdStrike/Cloud-AWS/tree/main/Security-Hub/install/sechub-identify-detections_lambda.zip

and when I extract I see three files.

gbs@prometheus-server:/ebs/crowdstrike/Cloud-AWS/Security-Hub/install

$ unzip sechub-identify-detections_lambda.zip Archive: sechub-identify-detections_lambda.zip extracting: init.py inflating: functions.py inflating: main.py

So functions.py is what you were referring here as lamda function(correct me If not) and create environment variable once the lambda is created.

jshcodes commented 1 year ago

Both files should be part of the lambda.

Once you've created the lambda and added both files, create the environment variables. πŸ‘

schikkam commented 1 year ago

Hi @jshcodes , I've deployed them them successfully.

I still see that it is throwing 401 unauthorized? Can you shed some light if there is a way to test it? I can see environment variables are set on lambda as well on node I'm executing the program.- They are valid.

image

jshcodes commented 1 year ago

I was reading the wrong documentation for the lambda config. We don't need the environment config for this one, my apologies.

We do still need Parameter Store though. We need to recreate the API key variables to be FIG_FALCON_CLIENT_ID and FIG_FALCON_CLIENT_SECRET instead.

https://github.com/CrowdStrike/Cloud-AWS/tree/main/Security-Hub#parameter-store-example

schikkam commented 1 year ago

I actually tried it using config file just to see if that works out. Created config.json file the fields required and started.

Now I see the log content as below:

Fri Jun  9 11:04:05 2023 Configuration parameters loaded from local file

Fri Jun  9 11:04:06 2023 Failed to connect to the API

Fri Jun  9 11:04:06 2023 Process terminated

Here is my config.file:

{
    "base_url":"https://api.laggar.gcw.crowdstrike.com/",
    "falcon_client_id":"XXXXXXX",
    "falcon_client_secret":"XXXXX",
    "app_id":"FIG_APP_ID",
    "severity_threshold":3,
    "sqs_queue_name":"SQS_QUEUE_NAME_SAME_REGION_AS_BELOW",
    "region":"US-GOV-1",
    "confirm_provider": true,
    "ssl_verify": true
}

What is the app_id should it be? Is this just a name ? Also, base_url has been set based on what I see on Falcon Mgmt Console.

schikkam commented 1 year ago

Just fyi that I see that it has access:

[fig@prometheus-server fig]$ nc -v api.laggar.gcw.crowdstrike.com 443
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 3.30.70.111:443.
^C
jshcodes commented 1 year ago

I actually tried it using config file just to see if that works out. Created config.json file the fields required and started.

Now I see the log content as below:

Fri Jun  9 11:04:05 2023 Configuration parameters loaded from local file

Fri Jun  9 11:04:06 2023 Failed to connect to the API

Fri Jun  9 11:04:06 2023 Process terminated

Here is my config.file:

{
    "base_url":"https://api.laggar.gcw.crowdstrike.com/",
    "falcon_client_id":"XXXXXXX",
    "falcon_client_secret":"XXXXX",
    "app_id":"FIG_APP_ID",
    "severity_threshold":3,
    "sqs_queue_name":"SQS_QUEUE_NAME_SAME_REGION_AS_BELOW",
    "region":"US-GOV-1",
    "confirm_provider": true,
    "ssl_verify": true
}

What is the app_id should it be? Is this just a name ? Also, base_url has been set based on what I see on Falcon Mgmt Console.

Note: Region in the config file should be pointing at the AWS region the SQS queue resides in.

schikkam commented 1 year ago

There were issues with attributes defined in config file as they aren't matching in code. Hence I've hard coded them in source and started , Now the good news is that it is able to authenticate but failed at SQS related. Have created sample SQS and started, here is the error I landed on now:

[fig@prometheus-server fig]$ /usr/bin/python3 /usr/share/fig/main.py
Traceback (most recent call last):
  File "/usr/share/fig/main.py", line 167, in <module>
    startStreaming(new_streams["body"]["resources"], current_cid)
  File "/usr/share/fig/main.py", line 57, in startStreaming
    for active_stream in new_streams:
TypeError: 'NoneType' object is not iterable

Please share your thoughts! 
jshcodes commented 1 year ago

There were issues with attributes defined in config file as they aren't matching in code. Hence I've hard coded them in source and started , Now the good news is that it is able to authenticate but failed at SQS related. Have created sample SQS and started, here is the error I landed on now:

[fig@prometheus-server fig]$ /usr/bin/python3 /usr/share/fig/main.py
Traceback (most recent call last):
  File "/usr/share/fig/main.py", line 167, in <module>
    startStreaming(new_streams["body"]["resources"], current_cid)
  File "/usr/share/fig/main.py", line 57, in startStreaming
    for active_stream in new_streams:
TypeError: 'NoneType' object is not iterable
Please share your thoughts! 

Check and make sure your API key has the following scopes:

schikkam commented 1 year ago

Yeah. To be sure I'd given full privileges to it - as I'm debugging.

schikkam commented 1 year ago

I tried to pull data using those secrets using https://assets.falcon.laggar.gcw.crowdstrike.com/support/api/swagger-eagle.html#/ and it works perfect. --fyi

jshcodes commented 1 year ago

You're not getting a stream back. Since we've hardcoded values, I'm not sure exactly what to check, but ideally we print the API call to the screen (by editing the same main.py file) to get an idea for what is causing the failure. Something like...

new_streams = falcon.command(action="listAvailableStreamsOAuth2", appId=config["app_id"])
raise SystemExit(new_streams)

... should spit out the entire API response for the request to list the streams.

Something else strange: The line numbers in your error message above don't match the current code base. Do you happen to know which version of the installer you ran?

schikkam commented 1 year ago

The only lines I changed is just this,.

falcon = FalconSDK.APIHarness(creds={'client_id': "XXXXX",
                                     'client_secret': "XXXXXXXXXXX"
                                     }, base_url="https://api.laggar.gcw.crowdstrike.com")
gbs@prometheus-server:/ebs/crowdstrike/Cloud-AWS/Security-Hub/install

$ ls -l
total 280
-rwxr-x---. 1 gbs gbp 24855 Jun  2 12:56 fig-2.0.19-install.run
-rw-r--r--. 1 gbs gbs  7196 Jul 31  2021 functions.py
-rw-r--r--. 1 gbs gbs     0 Jul 31  2021 __init__.py
-rwxr-x---. 1 gbs gbs   323 Jun  6 08:54 ins.sh
-rwxr-x---. 1 gbs gbp   911 Jun  2 12:56 install.sh
-rw-r--r--. 1 gbs gbs  2423 Jul 31  2021 main.py
-rw-r-----. 1 gbs gbp   911 Jun  2 12:56 README.md
-rwxr-x---. 1 gbs gbp 24861 Jun  2 12:56 sechub-2.0.21-install.run
-rwxr-x---. 1 gbs gbp 25003 Jun  2 12:56 sechub-2.0.22-install.run
-rwxr-x---. 1 gbs gbp 25064 Jun  2 12:56 sechub-2.0.23-install.run
-rwxr-x---. 1 gbs gbp 27169 Jun  2 12:56 sechub-2.0.32-install.run
-rwxr-x---. 1 gbs gbp 23461 Jun  2 12:56 sechub-2.0.33-install.run
-rwxr-x---. 1 gbs gbp 27175 Jun  2 12:56 sechub-2.0.34-install.run
-rwxr-x---. 1 gbs gbp 27300 Jun  2 12:56 sechub-2.0.36-install.run
-rwxr-x---. 1 gbs gbp 27300 Jun  2 12:56 sechub-2.0.latest-install.run
-rw-r-----. 1 gbs gbp  3786 Jun  2 12:56 sechub-identify-detections_lambda.zip
-rw-r-----. 1 gbs gbp   335 Jun  2 12:56 userdata.sh
jshcodes commented 1 year ago

The only lines I changed is just this,.

falcon = FalconSDK.APIHarness(creds={'client_id': "XXXXX",
                                     'client_secret': "XXXXXXXXXXX"
                                     }, base_url="https://api.laggar.gcw.crowdstrike.com")
gbs@prometheus-server:/ebs/crowdstrike/Cloud-AWS/Security-Hub/install

$ ls -l
total 280
-rwxr-x---. 1 gbs gbp 24855 Jun  2 12:56 fig-2.0.19-install.run
-rw-r--r--. 1 gbs gbs  7196 Jul 31  2021 functions.py
-rw-r--r--. 1 gbs gbs     0 Jul 31  2021 __init__.py
-rwxr-x---. 1 gbs gbs   323 Jun  6 08:54 ins.sh
-rwxr-x---. 1 gbs gbp   911 Jun  2 12:56 install.sh
-rw-r--r--. 1 gbs gbs  2423 Jul 31  2021 main.py
-rw-r-----. 1 gbs gbp   911 Jun  2 12:56 README.md
-rwxr-x---. 1 gbs gbp 24861 Jun  2 12:56 sechub-2.0.21-install.run
-rwxr-x---. 1 gbs gbp 25003 Jun  2 12:56 sechub-2.0.22-install.run
-rwxr-x---. 1 gbs gbp 25064 Jun  2 12:56 sechub-2.0.23-install.run
-rwxr-x---. 1 gbs gbp 27169 Jun  2 12:56 sechub-2.0.32-install.run
-rwxr-x---. 1 gbs gbp 23461 Jun  2 12:56 sechub-2.0.33-install.run
-rwxr-x---. 1 gbs gbp 27175 Jun  2 12:56 sechub-2.0.34-install.run
-rwxr-x---. 1 gbs gbp 27300 Jun  2 12:56 sechub-2.0.36-install.run
-rwxr-x---. 1 gbs gbp 27300 Jun  2 12:56 sechub-2.0.latest-install.run
-rw-r-----. 1 gbs gbp  3786 Jun  2 12:56 sechub-identify-detections_lambda.zip
-rw-r-----. 1 gbs gbp   335 Jun  2 12:56 userdata.sh

Did you execute sechub-2.0.latest-install.run?

schikkam commented 1 year ago

No I didn't Just this "fig-2.0.19-install.run"

jshcodes commented 1 year ago

No I didn't Just this "fig-2.0.19-install.run"

Rerun the install using the latest installer. Maybe this is a source version issue.

schikkam commented 1 year ago

Stared it with new installer and it is installed:


gbs@prometheus-server:/ebs/crowdstrike/Cloud-AWS/Security-Hub/install

$ sudo ./sechub-2.0.21-install.run --target /usr/share/fig
Creating directory /usr/share/fig
Verifying archive integrity...  100%   MD5 checksums are OK. All good.
Uncompressing Falcon Integration Gateway 2.0  100%
Creating service user
groupadd: group 'fig' already exists
adduser: user 'fig' already exists
Installing pre-requisites
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2-core                                                                                                                                                                                                                                              | 3.7 kB  00:00:00
amzn2extra-docker                                                                                                                                                                                                                                       | 3.0 kB  00:00:00
amzn2extra-epel                                                                                                                                                                                                                                         | 3.0 kB  00:00:00
amzn2extra-kernel-5.10                                                                                                                                                                                                                                  | 3.0 kB  00:00:00
epel/x86_64/metalink                                                                                                                                                                                                                                    |  24 kB  00:00:00
224 packages excluded due to repository priority protections
Package python3-3.7.16-1.amzn2.0.2.x86_64 already installed and latest version
Nothing to do
Requirement already satisfied: crowdstrike-falconpy in /home/fig/.local/lib/python3.7/site-packages (1.2.15)
Requirement already satisfied: requests in /home/fig/.local/lib/python3.7/site-packages (from crowdstrike-falconpy) (2.31.0)
Requirement already satisfied: urllib3 in /home/fig/.local/lib/python3.7/site-packages (from crowdstrike-falconpy) (1.26.16)
Requirement already satisfied: certifi>=2017.4.17 in /home/fig/.local/lib/python3.7/site-packages (from requests->crowdstrike-falconpy) (2023.5.7)
Requirement already satisfied: idna<4,>=2.5 in /home/fig/.local/lib/python3.7/site-packages (from requests->crowdstrike-falconpy) (3.4)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/fig/.local/lib/python3.7/site-packages (from requests->crowdstrike-falconpy) (3.1.0)
Requirement already satisfied: boto3 in /home/fig/.local/lib/python3.7/site-packages (1.26.147)
Requirement already satisfied: botocore<1.30.0,>=1.29.147 in /home/fig/.local/lib/python3.7/site-packages (from boto3) (1.29.147)
Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /home/fig/.local/lib/python3.7/site-packages (from boto3) (0.6.1)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/fig/.local/lib/python3.7/site-packages (from boto3) (1.0.1)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/fig/.local/lib/python3.7/site-packages (from botocore<1.30.0,>=1.29.147->boto3) (2.8.2)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/fig/.local/lib/python3.7/site-packages (from botocore<1.30.0,>=1.29.147->boto3) (1.26.16)
Requirement already satisfied: six>=1.5 in /home/fig/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.30.0,>=1.29.147->boto3) (1.16.0)
Setting permissions
Installing service
Cleaning up
Installation complete

Status:


gbs@prometheus-server:/ebs/crowdstrike/Cloud-AWS/Security-Hub/install

$ sudo systemctl status  fig
● fig.service - Security Hub Integration
   Loaded: loaded (/usr/lib/systemd/system/fig.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2023-06-09 12:56:34 UTC; 53ms ago
 Main PID: 3300 (python3)
   CGroup: /system.slice/fig.service
           └─3300 /usr/bin/python3 /usr/share/fig/main.py &> /dev/null

It didn't solve, same error as old with new installer


[fig@prometheus-server fig]$ /usr/bin/python3 /usr/share/fig/main.py
Traceback (most recent call last):
  File "/usr/share/fig/main.py", line 168, in <module>
    startStreaming(new_streams["body"]["resources"], current_cid)
  File "/usr/share/fig/main.py", line 57, in startStreaming
    for active_stream in new_streams:
TypeError: 'NoneType' object is not iterable
[fig@prometheus-server fig]$```
schikkam commented 1 year ago

It is started and then died. So manually started and I see this error

schikkam commented 1 year ago

This is the clone I have : https://github.com/CrowdStrike/Cloud-AWS

jshcodes commented 1 year ago

At this point, we will need to do the screen output suggestion from above so we can get an idea of what the API is responding with. Add raise SystemExit(new_streams) as a new line on line #185, right after this statement on line #184. And then review the API response for potential culprits.

schikkam commented 1 year ago

Hi @jshcodes - The code base looks different as I don't see that many lines. It just has 179 lines.

[fig@prometheus-server ~]$ wc -l /usr/share/fig/main.py
179 /usr/share/fig/main.py
[fig@prometheus-server ~]$

Please let me know if this clone is not the right one https://github.com/CrowdStrike/Cloud-AWS/issues/237#issuecomment-1584554396

jshcodes commented 1 year ago

Hi @jshcodes - The code base looks different as I don't see that many lines. It just has 179 lines.

[fig@prometheus-server ~]$ wc -l /usr/share/fig/main.py
179 /usr/share/fig/main.py
[fig@prometheus-server ~]$

Please let me know if this clone is not the right one #237 (comment)

I think you need to re-pull, or re-clone the repo. The current version of the file has 208 lines: https://github.com/CrowdStrike/Cloud-AWS/blob/main/Security-Hub/main.py

schikkam commented 1 year ago

Looks like the installer is not pulling the right version of that file when I install. I don't get that file you were referring here. Should I try copying that file and execute ? image

It is actually become a blocker for one of the customer. Is it possible to show you live if there is a real issue with-in the code or not! Appreciate your response on this!

carlosmmatos commented 1 year ago

@schikkam I see some sudo / permission denied errors being thrown around there.. I just ran this and here is the output in the event you would like to see if my commands work for you. Preferably this should be done on a new instance that's nice and clean:

[ansible@ip-172-31-22-x ~]$ FigFileName=sechub-2.0.latest-install.run
[ansible@ip-172-31-22-x ~]$ cd /var/tmp
[ansible@ip-172-31-22-x tmp]$ wget -O ${FigFileName} https://raw.githubusercontent.com/CrowdStrike/Cloud-AWS/master/Security-Hub/install/${FigFileName}
--2023-06-15 13:34:12--  https://raw.githubusercontent.com/CrowdStrike/Cloud-AWS/master/Security-Hub/install/sechub-2.0.latest-install.run
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27300 (27K) [application/octet-stream]
Saving to: β€˜sechub-2.0.latest-install.run’

100%[==================================================================================================================================================================================================>] 27,300      --.-K/s   in 0.004s

2023-06-15 13:34:12 (5.96 MB/s) - β€˜sechub-2.0.latest-install.run’ saved [27300/27300]

[ansible@ip-172-31-22-x tmp]$ chmod 755 ${FigFileName}
[ansible@ip-172-31-22-x tmp]$ # Run either as root or with sudo #^C
[ansible@ip-172-31-22-x tmp]$ sudo ./sechub-2.0.latest-install.run --target /usr/share/fig
Creating directory /usr/share/fig
Verifying archive integrity...  100%   MD5 checksums are OK. All good.
Uncompressing Falcon Integration Gateway 2.0  100%
Creating service user
Installing pre-requisites
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2-core                                                                                                                                                                                                           | 3.7 kB  00:00:00
Package python3-3.7.16-1.amzn2.0.2.x86_64 already installed and latest version
Nothing to do
Collecting crowdstrike-falconpy
  Downloading crowdstrike_falconpy-1.2.16-py3-none-any.whl (450 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 450 kB 9.8 MB/s
Collecting urllib3
  Downloading urllib3-2.0.3-py3-none-any.whl (123 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 123 kB 37.2 MB/s
Collecting requests
  Downloading requests-2.31.0-py3-none-any.whl (62 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 62 kB 2.6 MB/s
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.1.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (171 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 171 kB 36.5 MB/s
Collecting idna<4,>=2.5
  Downloading idna-3.4-py3-none-any.whl (61 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 61 kB 251 kB/s
Collecting certifi>=2017.4.17
  Downloading certifi-2023.5.7-py3-none-any.whl (156 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 156 kB 37.0 MB/s
Installing collected packages: urllib3, charset-normalizer, idna, certifi, requests, crowdstrike-falconpy
  WARNING: The script normalizer is installed in '/home/fig/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed certifi-2023.5.7 charset-normalizer-3.1.0 crowdstrike-falconpy-1.2.16 idna-3.4 requests-2.31.0 urllib3-2.0.3
Collecting boto3
  Downloading boto3-1.26.153-py3-none-any.whl (135 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 135 kB 9.1 MB/s
Collecting jmespath<2.0.0,>=0.7.1
  Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting botocore<1.30.0,>=1.29.153
  Downloading botocore-1.29.153-py3-none-any.whl (10.9 MB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10.9 MB 38.3 MB/s
Collecting s3transfer<0.7.0,>=0.6.0
  Downloading s3transfer-0.6.1-py3-none-any.whl (79 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 79 kB 14.9 MB/s
Collecting python-dateutil<3.0.0,>=2.1
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 247 kB 37.1 MB/s
Collecting urllib3<1.27,>=1.25.4
  Downloading urllib3-1.26.16-py2.py3-none-any.whl (143 kB)
     |β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 143 kB 37.5 MB/s
Collecting six>=1.5
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: jmespath, six, python-dateutil, urllib3, botocore, s3transfer, boto3
  Attempting uninstall: urllib3
    Found existing installation: urllib3 2.0.3
    Uninstalling urllib3-2.0.3:
      Successfully uninstalled urllib3-2.0.3
Successfully installed boto3-1.26.153 botocore-1.29.153 jmespath-1.0.1 python-dateutil-2.8.2 s3transfer-0.6.1 six-1.16.0 urllib3-1.26.16
Setting permissions
Installing service
Created symlink from /etc/systemd/system/multi-user.target.wants/fig.service to /usr/lib/systemd/system/fig.service.
Cleaning up
Installation complete
[ansible@ip-172-31-22-x tmp]$ cat /usr/lib/systemd/system/fig.service
[Unit]
Description=Security Hub Integration
After=multi-user.target

[Service]
WorkingDirectory=/usr/share/fig
User=fig
Type=idle
ExecStart=/usr/bin/python3 /usr/share/fig/main.py &> /dev/null
Restart=always

[Install]
WantedBy=multi-user.target
[ansible@ip-172-31-22-x tmp]$

You shouldn't have to run the ./install.sh script. The steps I'm using are basically a mirror of the Security-Hub/install/userdata.sh script.

If you are still seeing permission denied errors with sudo: then you probably might need to check your sudo settings for the user who is executing the script. If possible, and if you have access, I would try to run it as the root use to rule anything else out as well. Hope this helps.

schikkam commented 1 year ago

Hi @carlosmmatos , I've Executed all those steps as listed as root user but I still don't see working.
Here is the sequence of command executed:


root@prometheus-server:/var/tmp

$ FigFileName=sechub-2.0.latest-install.run

root@prometheus-server:/var/tmp

$ wget -O ${FigFileName} https://raw.githubusercontent.com/CrowdStrike/Cloud-AWS/master/Security-Hub/install/${FigFileName}
--2023-06-15 16:48:11--  https://raw.githubusercontent.com/CrowdStrike/Cloud-AWS/master/Security-Hub/install/sechub-2.0.latest-install.run
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 27300 (27K) [application/octet-stream]
Saving to: β€˜sechub-2.0.latest-install.run’

100%[=========================================================================================================================================================================>] 27,300      --.-K/s   in 0s

2023-06-15 16:48:11 (70.2 MB/s) - β€˜sechub-2.0.latest-install.run’ saved [27300/27300]

root@prometheus-server:/var/tmp

$ chmod 755 ${FigFileName}

root@prometheus-server:/var/tmp

$ sudo sh  ./sechub-2.0.latest-install.run --target /usr/share/fig
Creating directory /usr/share/fig
Verifying archive integrity...  100%   MD5 checksums are OK. All good.
Uncompressing Falcon Integration Gateway 2.0  100%
Creating service user
adduser: warning: the home directory already exists.
Not copying any file from skel directory into it.
Creating mailbox file: File exists
Installing pre-requisites
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2-core                                                                                                                                                                                  | 3.7 kB  00:00:00
224 packages excluded due to repository priority protections
Package python3-3.7.16-1.amzn2.0.2.x86_64 already installed and latest version
Nothing to do
Requirement already satisfied: crowdstrike-falconpy in /home/fig/.local/lib/python3.7/site-packages (1.2.15)
Requirement already satisfied: requests in /home/fig/.local/lib/python3.7/site-packages (from crowdstrike-falconpy) (2.31.0)
Requirement already satisfied: urllib3 in /home/fig/.local/lib/python3.7/site-packages (from crowdstrike-falconpy) (1.26.16)
Requirement already satisfied: idna<4,>=2.5 in /home/fig/.local/lib/python3.7/site-packages (from requests->crowdstrike-falconpy) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /home/fig/.local/lib/python3.7/site-packages (from requests->crowdstrike-falconpy) (2023.5.7)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/fig/.local/lib/python3.7/site-packages (from requests->crowdstrike-falconpy) (3.1.0)
Requirement already satisfied: boto3 in /home/fig/.local/lib/python3.7/site-packages (1.26.147)
Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /home/fig/.local/lib/python3.7/site-packages (from boto3) (0.6.1)
Requirement already satisfied: botocore<1.30.0,>=1.29.147 in /home/fig/.local/lib/python3.7/site-packages (from boto3) (1.29.147)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/fig/.local/lib/python3.7/site-packages (from boto3) (1.0.1)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/fig/.local/lib/python3.7/site-packages (from botocore<1.30.0,>=1.29.147->boto3) (2.8.2)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/fig/.local/lib/python3.7/site-packages (from botocore<1.30.0,>=1.29.147->boto3) (1.26.16)
Requirement already satisfied: six>=1.5 in /home/fig/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.30.0,>=1.29.147->boto3) (1.16.0)
Setting permissions
Installing service
Cleaning up
Installation complete

root@prometheus-server:/var/tmp

$ cat /usr/lib/systemd/system/fig.service
[Unit]
Description=Security Hub Integration
After=multi-user.target

[Service]
WorkingDirectory=/usr/share/fig
User=fig
Type=idle
ExecStart=/usr/bin/python3 /usr/share/fig/main.py &> /dev/null
Restart=always

[Install]
WantedBy=multi-user.target

root@prometheus-server:/var/tmp

$ wc -l /usr/share/fig/main.py
208 /usr/share/fig/main.py

root@prometheus-server:/var/tmp

$ systemctl start   fig

root@prometheus-server:/var/tmp

$ systemctl status  fig
● fig.service - Security Hub Integration
   Loaded: loaded (/usr/lib/systemd/system/fig.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Thu 2023-06-15 16:50:06 UTC; 6s ago
  Process: 22435 ExecStart=/usr/bin/python3 /usr/share/fig/main.py &> /dev/null (code=exited, status=1/FAILURE)
 Main PID: 22435 (code=exited, status=1/FAILURE)

root@prometheus-server:/var/tmp

$ tail -f /usr/share/fi
fig/       file/      firstboot/

root@prometheus-server:/var/tmp

$ tail -f /usr/share/fi
fig/       file/      firstboot/

root@prometheus-server:/var/tmp

$ tail -f /usr/share/fig/fig-service.log
Thu Jun 15 16:50:05 2023 Unable to load configuration parameters.

Thu Jun 15 16:50:05 2023 Process terminated

Thu Jun 15 16:50:05 2023 Specified configuration file not found

Thu Jun 15 16:50:05 2023 Unable to load configuration parameters.

Thu Jun 15 16:50:05 2023 Process terminated

^C

root@prometheus-server:/var/tmp

$ date
Thu Jun 15 16:50:30 UTC 2023

root@prometheus-server:/var/tmp

$
schikkam commented 1 year ago

Tried to run with config.json (copied with id/secrets) and I see this error as below


gbs@prometheus-server:/ebs/crowdstrike/Cloud-AWS/Security-Hub

[fig@prometheus-server fig]$ nc -v api.laggar.gcw.crowdstrike.com 443
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 15.200.196.193:443.
^C
[fig@prometheus-server fig]$ /usr/bin/python3 /usr/share/fig/main.py
Failed to connect to the API on https://api.laggar.gcw.crowdstrike.com/.  Check base_url and ssl_verify configuration settings.
[fig@prometheus-server fig]$ tail -f fig-service.log
Thu Jun 15 17:01:45 2023 Failed to connect to the API on https://api.laggar.gcw.crowdstrike.com/. Check base_url and ssl_verify configuration settings.

Thu Jun 15 17:01:45 2023 Process terminated

Thu Jun 15 17:02:27 2023 Configuration parameters loaded from local file

Thu Jun 15 17:02:28 2023 Failed to connect to the API on https://api.laggar.gcw.crowdstrike.com/. Check base_url and ssl_verify configuration settings.

Thu Jun 15 17:02:28 2023 Process terminated

config.json

[fig@prometheus-server fig]$ cat config.json
{
    "api_base_url":"https://api.laggar.gcw.crowdstrike.com/",
    "falcon_client_id":"XXX",
    "falcon_client_secret":"XXXX",
    "app_id":"myapp",
    "severity_threshold":3,
    "sqs_queue_name":"SQS_QUEUE_NAME_SAME_REGION_AS_BELOW",
    "region":"us-east-1",
    "confirm_provider": true,
    "ssl_verify": true
}

[fig@prometheus-server fig]$

I've tried ssl_verify - false to see if that works but still failed with same error.

Please let me know!

schikkam commented 1 year ago

I do have proper access to that :

[fig@prometheus-server fig]$ nc -v api.laggar.gcw.crowdstrike.com 443
Ncat: Version 7.50 ( https://nmap.org/ncat )
Ncat: Connected to 3.30.211.25:443.
carlosmmatos commented 1 year ago

@schikkam Hey sorry for the late response, not sure if this was fixed but something just popped up at me with this issue. Can you do the following change in your config.json:

- "api_base_url":"https://api.laggar.gcw.crowdstrike.com/",
+ "api_base_url":"https://api.laggar.gcw.crowdstrike.com",

Remove the trailing /

schikkam commented 1 year ago

Thanks. It didn't work. Is is possible to do a live call and show you, so that it is easy for us to know if there is a problem with-in core? Appreciate your help.

carlosmmatos commented 1 year ago

@schikkam open an issue via the support portal and reference this GitHub issue in it. The ticket should be routed to us and we can schedule something to help get this situated for you.

carlosmmatos commented 11 months ago

@schikkam check out the newly released manual install guide. I would also use the latest installer as well.

carlosmmatos commented 8 months ago

Closing due to inactivity. If you would like to re-open please either re-open this issue or create a new one. Thanks.