praetorian-inc / gato

GitHub Actions Pipeline Enumeration and Attack Tool
Apache License 2.0
561 stars 47 forks source link

Crash w/ Stack Trace and None Type Error when Company PAT vs Personal PAT #62

Closed json-yo closed 6 months ago

json-yo commented 9 months ago

I ran gato against a target repo with a 3rd party PAT (i.e. personal one) and it worked fine for enumeration. I then used a PAT for my work (classic PAT like before) GH account and when I tried to enumerate, it throws a NoneType error complaining about it not being subscriptable.

Additional Info:

gato -s e -t XXXXX --output-yaml . --output-json XXXXX.json [+] The authenticated user is: XXXX [!] The token has no scopes! [+] Enumerating the XXXXX organization! [!] The user has only public access! [+] About to enumerate 379 repos within the XXXXX organization! [+] Querying and caching workflow YAML files! Traceback (most recent call last): File "/root/gato/venv/bin/gato", line 8, in sys.exit(entry()) File "/root/gato/venv/lib/python3.10/site-packages/gato/main.py", line 6, in entry sys.exit(cli.cli(sys.argv[1:])) File "/root/gato/venv/lib/python3.10/site-packages/gato/cli/cli.py", line 83, in cli arguments.func(arguments, subparsers) File "/root/gato/venv/lib/python3.10/site-packages/gato/cli/cli.py", line 242, in enumerate orgs = [gh_enumeration_runner.enumerate_organization( File "/root/gato/venv/lib/python3.10/site-packages/gato/enumerate/enumerate.py", line 184, in enumerate_organization self.repo_e.construct_workflow_cache(result.json()['data']['nodes']) File "/root/gato/venv/lib/python3.10/site-packages/gato/enumerate/repository.py", line 190, in construct_workflow_cache owner = result['nameWithOwner'] TypeError: 'NoneType' object is not subscriptable

AdnaneKhan commented 9 months ago

I ran gato against a target repo with a 3rd party PAT (i.e. personal one) and it worked fine for enumeration. I then used a PAT for my work (classic PAT like before) GH account and when I tried to enumerate, it throws a NoneType error complaining about it not being subscriptable.

Additional Info:

gato -s e -t XXXXX --output-yaml . --output-json XXXXX.json [+] The authenticated user is: XXXX [!] The token has no scopes! [+] Enumerating the XXXXX organization! [!] The user has only public access! [+] About to enumerate 379 repos within the XXXXX organization! [+] Querying and caching workflow YAML files! Traceback (most recent call last): File "/root/gato/venv/bin/gato", line 8, in sys.exit(entry()) File "/root/gato/venv/lib/python3.10/site-packages/gato/main.py", line 6, in entry sys.exit(cli.cli(sys.argv[1:])) File "/root/gato/venv/lib/python3.10/site-packages/gato/cli/cli.py", line 83, in cli arguments.func(arguments, subparsers) File "/root/gato/venv/lib/python3.10/site-packages/gato/cli/cli.py", line 242, in enumerate orgs = [gh_enumeration_runner.enumerate_organization( File "/root/gato/venv/lib/python3.10/site-packages/gato/enumerate/enumerate.py", line 184, in enumerate_organization self.repo_e.construct_workflow_cache(result.json()['data']['nodes']) File "/root/gato/venv/lib/python3.10/site-packages/gato/enumerate/repository.py", line 190, in construct_workflow_cache owner = result['nameWithOwner'] TypeError: 'NoneType' object is not subscriptable

Ah, I've seen a similar issue while working on a newer feature. The GraphQL API can be a bit weird with what it returns (it'll be a 200 but the expected fields might not be present).

Will have a patch in to main and dev when I get some time. If you need this to work in the interim, try adding the following to construct_workflow_cache in repository.py:

Old:

    for result in yml_results:
            owner = result['nameWithOwner']

            self.workflow_cache[owner] = list()

            if not result['object']:
                continue

New:

        for result in yml_results:
            # If we get any malformed/missing data just skip it and 
            # Gato will fall back to the contents API for these few cases.
            if not result:
                continue

            if 'nameWithOwner' not in result:
                continue

            owner = result['nameWithOwner']
            # Empty means no yamls, so just skip.
            if not result['object']:
                self.workflow_cache[owner] = list()
                continue
AdnaneKhan commented 9 months ago

@json-yo can you check out fix/result_error_handling and run gato? I'm curious if you run into the same issue.

DhiyaneshGeek commented 9 months ago

Hi @AdnaneKhan i checked out the new branch fix/result_error_handling

[+] Querying and caching workflow YAML files!
    Traceback (most recent call last):
  File "/Users/geekfreak/gato/venv/bin/gato", line 8, in <module>
    sys.exit(entry())
             ^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/main.py", line 6, in entry
    sys.exit(cli.cli(sys.argv[1:]))
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/cli/cli.py", line 83, in cli
    arguments.func(arguments, subparsers)
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/cli/cli.py", line 242, in enumerate
    orgs = [gh_enumeration_runner.enumerate_organization(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/enumerate/enumerate.py", line 184, in enumerate_organization
    self.repo_e.construct_workflow_cache(result.json()['data']['nodes'])
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/enumerate/repository.py", line 190, in construct_workflow_cache
    owner = result['nameWithOwner']
            ~~~~~~^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable
(venv) geekfreak@localhost gato % git status
On branch fix/result_error_handling
Your branch is up to date with 'origin/fix/result_error_handling'.

nothing to commit, working tree clean

still not working :( it crashes

AdnaneKhan commented 9 months ago

Hi @AdnaneKhan i checked out the new branch fix/result_error_handling

[+] Querying and caching workflow YAML files!
  Traceback (most recent call last):
  File "/Users/geekfreak/gato/venv/bin/gato", line 8, in <module>
    sys.exit(entry())
             ^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/main.py", line 6, in entry
    sys.exit(cli.cli(sys.argv[1:]))
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/cli/cli.py", line 83, in cli
    arguments.func(arguments, subparsers)
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/cli/cli.py", line 242, in enumerate
    orgs = [gh_enumeration_runner.enumerate_organization(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/enumerate/enumerate.py", line 184, in enumerate_organization
    self.repo_e.construct_workflow_cache(result.json()['data']['nodes'])
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/enumerate/repository.py", line 190, in construct_workflow_cache
    owner = result['nameWithOwner']
            ~~~~~~^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable
(venv) geekfreak@localhost gato % git status
On branch fix/result_error_handling
Your branch is up to date with 'origin/fix/result_error_handling'.

nothing to commit, working tree clean

still not working :( it crashes

Can you confirm you are running the updated code? Line 190 in the new branch is different.

You might need to clear your virtual environment or install it in editable mode (pip install -e .)

DhiyaneshGeek commented 9 months ago

Hi @AdnaneKhan

Checkout fix/result_error_handling

geekfreak@localhost gato % git checkout fix/result_error_handling
Already on 'fix/result_error_handling'
Your branch is up to date with 'origin/fix/result_error_handling'.

Git Pull

geekfreak@localhost gato % git pull
Already up to date.

activiate python variable

geekfreak@localhost gato % python3 -m venv venv
geekfreak@localhost gato % source venv/bin/activate

pip installing the new changes

(venv) geekfreak@localhost gato % pip install .
Processing /Users/geekfreak/gato
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: colorama in ./venv/lib/python3.11/site-packages (from praetorian-gato==1.6.0) (0.4.6)
Requirement already satisfied: requests in ./venv/lib/python3.11/site-packages (from praetorian-gato==1.6.0) (2.31.0)
Requirement already satisfied: pyyaml in ./venv/lib/python3.11/site-packages (from praetorian-gato==1.6.0) (6.0.1)
Requirement already satisfied: packaging in ./venv/lib/python3.11/site-packages (from praetorian-gato==1.6.0) (23.2)
Requirement already satisfied: cryptography in ./venv/lib/python3.11/site-packages (from praetorian-gato==1.6.0) (41.0.7)
Requirement already satisfied: cffi>=1.12 in ./venv/lib/python3.11/site-packages (from cryptography->praetorian-gato==1.6.0) (1.16.0)
Requirement already satisfied: charset-normalizer<4,>=2 in ./venv/lib/python3.11/site-packages (from requests->praetorian-gato==1.6.0) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./venv/lib/python3.11/site-packages (from requests->praetorian-gato==1.6.0) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./venv/lib/python3.11/site-packages (from requests->praetorian-gato==1.6.0) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in ./venv/lib/python3.11/site-packages (from requests->praetorian-gato==1.6.0) (2023.11.17)
Requirement already satisfied: pycparser in ./venv/lib/python3.11/site-packages (from cffi>=1.12->cryptography->praetorian-gato==1.6.0) (2.21)
Building wheels for collected packages: praetorian-gato
  Building wheel for praetorian-gato (pyproject.toml) ... done
  Created wheel for praetorian-gato: filename=praetorian_gato-1.6.0-py3-none-any.whl size=57426 sha256=2400642f88d5ac715880969635c8892b91451d659f864516fbbf92b6696f3fc7
  Stored in directory: /private/var/folders/2j/ywbxjmdx3bbcvr3w6qzpml2w0000gn/T/pip-ephem-wheel-cache-opmwrog1/wheels/b4/94/5d/1b195bbaabbeb3f3bb83a9da998a355b46215bcdbccda43b37
Successfully built praetorian-gato
Installing collected packages: praetorian-gato
  Attempting uninstall: praetorian-gato
    Found existing installation: praetorian-gato 1.6.0
    Uninstalling praetorian-gato-1.6.0:
      Successfully uninstalled praetorian-gato-1.6.0
Successfully installed praetorian-gato-1.6.0

[notice] A new release of pip is available: 23.3.1 -> 23.3.2
[notice] To update, run: pip install --upgrade pip

Running Against a target Organization

(venv) geekfreak@localhost gato % gato enum --target paytm               
No 'GH_TOKEN' environment variable set! Please enter a GitHub PAT.

......

[+] The GitHub Classic PAT has the following scopes: gist, notifications, read:discussion, read:org, read:packages, read:public_key, read:repo_hook, read:user, repo
[+] Enumerating the paytm organization!
[!] The user has only public access!
Traceback (most recent call last):
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1096, in _validate_conn
    conn.connect()
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connection.py", line 642, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connection.py", line 782, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
               ^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 470, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 514, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1108, in _create
    self.do_handshake()
  File "/opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1379, in do_handshake
    self._sslobj.do_handshake()
TimeoutError: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 469, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.github.com', port=443): Read timed out. (read timeout=None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/geekfreak/gato/venv/bin/gato", line 8, in <module>
    sys.exit(entry())
             ^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/main.py", line 6, in entry
    sys.exit(cli.cli(sys.argv[1:]))
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/cli/cli.py", line 83, in cli
    arguments.func(arguments, subparsers)
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/cli/cli.py", line 242, in enumerate
    orgs = [gh_enumeration_runner.enumerate_organization(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/enumerate/enumerate.py", line 168, in enumerate_organization
    enum_list = self.org_e.construct_repo_enum_list(organization)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/enumerate/organization.py", line 47, in construct_repo_enum_list
    org_private_repos = self.__assemble_repo_list(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/enumerate/organization.py", line 30, in __assemble_repo_list
    raw_repos = self.api.check_org_repos(organization, visibility)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/github/api.py", line 550, in check_org_repos
    org_repos = self.call_get(f'/orgs/{org}/repos', params=get_params)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/github/api.py", line 216, in call_get
    api_response = requests.get(request_url, headers=get_header,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.github.com', port=443): Read timed out. (read timeout=None)

after that i got the above error

Let me know if i have missed out some steps

Thanks

mas0nd commented 6 months ago

The original issue with the workflow caching system has been fixed and merged to main. Please let us know if you run into any similar issues here or in a separate issue. Thanks!