search

chronicle / detection-rules

Collection of YARA-L 2.0 sample rules for the Chronicle Detection API
https://chronicle.security
Apache License 2.0
275 stars 66 forks source link

chronicle_auth.py doesn't read in dotenv environment variable #64

Closed Bacon-404 closed 1 month ago

Bacon-404 commented 2 months ago

Version 3.12.2 Arch: 64 Packages: annotated-types, 0.6.0 cachetools, 5.3.3 certifi 2024.2.2 charset-normalizer, 3.3.2 google-auth, 2.29.0 idna, 3.7 pip, 24.0 pyasn1, 0.6.0 pyasn1_module, 0.4.0 pydantic, 2.6.4 pydantic_core, 2.16.3 pydantic-dotenv, 1.0.1 PyYAML, 6.0.1 requests, 2.31.0 rsa, 4.9 ruamel.yaml, 0.18.6 ruamel.yaml.clib, 0.2.8 typing_extensions, 4.11.0 urllib, 2.2.1

(venv312) PS \\rule_manager_c> python -m rule_cli --pull-latest-rules
14-May-24 13:28:16 Eastern Daylight Time | INFO | <module> | Rule CLI started
14-May-24 13:28:16 Eastern Daylight Time | INFO | <module> | Attempting to pull latest version of all rules from Chronicle and update local files
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "\\venv312\rule_manager_c\rule_cli\__main__.py", line 407, in <module>
    pull_latest_rules()
  File "\\venv312\rule_manager_c\rule_cli\__main__.py", line 63, in pull_latest_rules
    http_session = initialize_http_session()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File venv312\rule_manager_c\rule_cli\__main__.py", line 53, in initialize_http_session
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "\\venv312\rule_manager_c\rule_cli\__main__.py", line 407, in <module>
    pull_latest_rules()
  File "\\venv312\rule_manager_c\rule_cli\__main__.py", line 63, in pull_latest_rules
    http_session = initialize_http_session()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\\venv312\rule_manager_c\rule_cli\__main__.py", line 53, in initialize_http_session
    os.environ["CHRONICLE_API_CREDENTIALS"]
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen os>", line 685, in __getitem__
KeyError: 'CHRONICLE_API_CREDENTIALS'
(venv312) PS \\rule_manager_c>

Additionally following the setup directions there is a .env file at the root of the rule_manager_c named ruleimport.env with all the correct fields from SecOps and GCP Instance IDs.

threat-punter commented 2 months ago

Can you rename your env file to .env and try running the command again?

Bacon-404 commented 2 months ago

Thank you! That worked but getting an issue with the information I have included with the env file as it does not decode properly.

(venv312) PS \\venv312\rule_manager_c> python -m rule_cli --pull-latest-rules
15-May-24 11:38:26 Eastern Daylight Time | INFO | <module> | Rule CLI started
15-May-24 11:38:26 Eastern Daylight Time | INFO | <module> | Attempting to pull latest version of all rules from Chronicle and update local files
ne 52, in initialize_http_session
    chronicle_api_credentials=json.loads(
                              ^^^^^^^^^^^
  File "C:\Program Files\Python\312\Lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\312\Lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\312\Lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1882 (char 1881)

Will attempt to debug on my own but here is the error in case if its an easy fix.

threat-punter commented 2 months ago

Is the service account key that you're using to authenticate to SecOps' REST API in JSON format on a single line in your .env file like this?

CHRONICLE_API_CREDENTIALS={"type":"service_account","project_id":"xxx","private_key_id":"xxx","private_key":"xxx","client_email":"xxx","client_id":"xxx","auth_uri":"xxx","token_uri":"xxx","auth_provider_x509_cert_url":"xxx","client_x509_cert_url":"xxx","universe_domain":"xxx"}
Bacon-404 commented 2 months ago

Yep exactly in that format. Except one discrepancy is that the end of the JSON the credentials provided to me do not contain a universe domain.

threat-punter commented 2 months ago

🤔 The error that you shared below tells me that a string value wasn't found for environment variable CHRONICLE_API_CREDENTIALS.

JSONDecodeError("Expecting value", s, err.value)

Is your .env file in the tools/rule_manager directory? Are you running the python -m rule_cli command from the rule_manager directory as well?

Bacon-404 commented 2 months ago

Yeah in this line, I censored some private info from my earlier log. but otherwise it is straight from the CLI of my machine. 

(venv312) PS \\venv312\rule_manager_c> python -m rule_cli --pull-latest-rules

Here is an ls dump of my environment.

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----         5/13/2024   2:01 PM                chronicle_api
d-----         5/10/2024  12:38 PM                reference_lists
d-----         5/10/2024  12:38 PM                rules
d-----         5/10/2024   4:24 PM                rule_cli
-a----         5/13/2024   2:33 PM           2632 .env
-a----          5/9/2024  11:12 AM           1869 .gitignore
-a----          5/9/2024  11:12 AM           4763 .gitlab-ci.yml
-a----          5/9/2024  11:12 AM           1066 CONTRIBUTING.md
-a----          5/9/2024  11:12 AM          22106 README.md
-a----          5/9/2024  11:12 AM            362 reference_list_config.yaml
-a----          5/9/2024  11:12 AM             70 requirements.txt
-a----          5/9/2024  11:12 AM             26 requirements_dev.txt
-a----          5/9/2024  11:12 AM            526 rule_config.yaml
threat-punter commented 1 month ago

Can you add a print(os.environ) statement before this line, run your command again, and check if your variables are set correctly? Don't leave this code in or share the output though :)

https://github.com/chronicle/detection-rules/blob/main/tools/rule_manager/rule_cli/__main__.py#L51

Bacon-404 commented 1 month ago

Seems my packages are a bit borked from this latest message.

\\venv312\rule_manager_c\rule_cli\__init__.py", line 20, in <module>
    import dotenv
ModuleNotFoundError: No module named 'dotenv'
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----         5/13/2024   2:00 PM                annotated_types
d-----         5/13/2024   2:00 PM                annotated_types-0.6.0.dist-info
d-----         5/13/2024   2:00 PM                cachetools
d-----         5/13/2024   2:00 PM                cachetools-5.3.3.dist-info
d-----         5/13/2024   2:00 PM                certifi
d-----         5/13/2024   2:00 PM                certifi-2024.2.2.dist-info
d-----         5/13/2024   2:00 PM                charset_normalizer
d-----         5/13/2024   2:00 PM                charset_normalizer-3.3.2.dist-info
d-----         5/13/2024   2:00 PM                dotenv
d-----         5/13/2024   2:00 PM                google
d-----         5/13/2024   2:00 PM                google_auth-2.29.0.dist-info
d-----         5/13/2024   2:00 PM                idna
d-----         5/13/2024   2:00 PM                idna-3.7.dist-info
d-----         5/13/2024   2:00 PM                pip
d-----         5/13/2024   1:56 PM                pip-24.0.dist-info
d-----         5/13/2024   2:00 PM                pyasn1
d-----         5/13/2024   2:00 PM                pyasn1-0.6.0.dist-info
d-----         5/13/2024   2:00 PM                pyasn1_modules
d-----         5/13/2024   2:00 PM                pyasn1_modules-0.4.0.dist-info
d-----         5/13/2024   2:00 PM                pydantic
d-----         5/13/2024   2:00 PM                pydantic-2.6.4.dist-info
d-----         5/13/2024   2:00 PM                pydantic_core
d-----         5/13/2024   2:00 PM                pydantic_core-2.16.3.dist-info
d-----         5/13/2024   2:00 PM                python_dotenv-1.0.1.dist-info
d-----         5/13/2024   2:00 PM                PyYAML-6.0.1.dist-info
d-----         5/13/2024   2:00 PM                requests
d-----         5/13/2024   2:00 PM                requests-2.31.0.dist-info
d-----         5/13/2024   2:00 PM                rsa
d-----         5/13/2024   2:00 PM                rsa-4.9.dist-info
d-----         5/13/2024   2:00 PM                ruamel
d-----         5/13/2024   2:00 PM                ruamel.yaml-0.18.6.dist-info
d-----         5/13/2024   2:00 PM                ruamel.yaml.clib-0.2.8.dist-info
d-----         5/13/2024   2:00 PM                typing_extensions-4.11.0.dist-info
d-----         5/13/2024   2:00 PM                urllib3
d-----         5/13/2024   2:00 PM                urllib3-2.2.1.dist-info
d-----         5/13/2024   2:00 PM                yaml
d-----         5/13/2024   2:00 PM                _yaml
d-----         5/13/2024   2:00 PM                __pycache__
-a----         5/13/2024   1:56 PM              0 pip-24.0.virtualenv
-a----         5/13/2024   2:00 PM         122293 typing_extensions.py
-a----         5/13/2024   2:00 PM         266752 _ruamel_yaml.cp312-win_amd64.pyd
-a----         5/13/2024   1:56 PM             18 _virtualenv.pth
-a----         5/13/2024   1:56 PM           4546 _virtualenv.py

Edit: Upon going into my python virtualenv and jumping into the interpreter and running print(os.environ) I didn't find the defined .env variables in the dump. I could find variables for python, and virtualenv so I know those are working.

threat-punter commented 1 month ago

Can you try deleting and recreating your virtual environment and doing a fresh install of the project requirements (pip install -r requirements.txt)?

Bacon-404 commented 1 month ago

Rebuilt the project as you asked. 

\venv312

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----         5/19/2024   1:49 PM                Lib
d-----          5/9/2024  11:12 AM                rule_manager
d-----         5/19/2024   1:49 PM                Scripts
-a----         5/19/2024   1:49 PM             42 .gitignore
-a----         5/19/2024   1:49 PM            302 pyvenv.cfg
\\venv312\rule_manager

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----          5/9/2024  11:12 AM                chronicle_api
d-----          5/9/2024  11:12 AM                reference_lists
d-----          5/9/2024  11:12 AM                rules
d-----          5/9/2024  11:12 AM                rule_cli
-a----          5/9/2024  11:12 AM           1869 .gitignore
-a----          5/9/2024  11:12 AM           4763 .gitlab-ci.yml
-a----          5/9/2024  11:12 AM           1066 CONTRIBUTING.md
-a----          5/9/2024  11:12 AM          22106 README.md
-a----          5/9/2024  11:12 AM            362 reference_list_config.yaml
-a----          5/9/2024  11:12 AM             70 requirements.txt
-a----          5/9/2024  11:12 AM             26 requirements_dev.txt
-a----          5/9/2024  11:12 AM            526 rule_config.yaml

(venv312) PS \\venv312\rule_manager> pip install -r requirements.txt
Installing collected packages: urllib3, typing-extensions, ruamel.yaml.clib, PyYAML, python-dotenv, pyasn1, idna, charset-normalizer, certifi, cachetools, annotated-types, ruamel.yaml, rsa, requests, pydantic-core, pyasn1-modules, pydantic, google-auth
Successfully installed PyYAML-6.0.1 annotated-types-0.6.0 cachetools-5.3.3 certifi-2024.2.2 charset-normalizer-3.3.2 google-auth-2.29.0 idna-3.7 pyasn1-0.6.0 pyasn1-modules-0.4.0 pydantic-2.6.4 pydantic-core-2.16.3 python-dotenv-1.0.1 requests-2.31.0 rsa-4.9 ruamel.yaml-0.18.6 ruamel.yaml.clib-0.2.8 typing-extensions-4.11.0 urllib3-2.2.1

Installing everything went without a hitch and used the credentials from before in a file named .env . However I still get an error reading my credentials. Which tells me that something is wrong with them but at this point in time I am not sure what. Is there an alternative way I can provide authentication to rule_cli? 

venv312\rule_manager> python -m rule_cli --pull-latest-rules
19-May-24 14:13:11 Eastern Daylight Time | INFO | <module> | Rule CLI started
19-May-24 14:13:11 Eastern Daylight Time | INFO | <module> | Attempting to pull latest version of all rules from Chronicle and update local files
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "\\venv312\rule_manager\rule_cli\__main__.py", line 407, in <module>
    pull_latest_rules()
  File "\\venv312\rule_manager\rule_cli\__main__.py", line 63, in pull_latest_rules
    http_session = initialize_http_session()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\\venv312\rule_manager\rule_cli\__main__.py", line 52, in initialize_http_session
    chronicle_api_credentials=json.loads(
                              ^^^^^^^^^^^
  File "C:\Program Files\Python\312\Lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\312\Lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python\312\Lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1882 (char 1881)
(venv312) PS \\venv312\rule_manager>
Bacon-404 commented 1 month ago

Was able to get the credentials corrected and I'm looking to solve some permission issues. But for now the issue is mine to resolve. Thank you for your help

threat-punter commented 1 month ago

That's great! I'm glad to hear it