Closed json-yo closed 6 months ago
I ran gato against a target repo with a 3rd party PAT (i.e. personal one) and it worked fine for enumeration. I then used a PAT for my work (classic PAT like before) GH account and when I tried to enumerate, it throws a NoneType error complaining about it not being subscriptable.
Additional Info:
gato -s e -t XXXXX --output-yaml . --output-json XXXXX.json [+] The authenticated user is: XXXX [!] The token has no scopes! [+] Enumerating the XXXXX organization! [!] The user has only public access! [+] About to enumerate 379 repos within the XXXXX organization! [+] Querying and caching workflow YAML files! Traceback (most recent call last): File "/root/gato/venv/bin/gato", line 8, in sys.exit(entry()) File "/root/gato/venv/lib/python3.10/site-packages/gato/main.py", line 6, in entry sys.exit(cli.cli(sys.argv[1:])) File "/root/gato/venv/lib/python3.10/site-packages/gato/cli/cli.py", line 83, in cli arguments.func(arguments, subparsers) File "/root/gato/venv/lib/python3.10/site-packages/gato/cli/cli.py", line 242, in enumerate orgs = [gh_enumeration_runner.enumerate_organization( File "/root/gato/venv/lib/python3.10/site-packages/gato/enumerate/enumerate.py", line 184, in enumerate_organization self.repo_e.construct_workflow_cache(result.json()['data']['nodes']) File "/root/gato/venv/lib/python3.10/site-packages/gato/enumerate/repository.py", line 190, in construct_workflow_cache owner = result['nameWithOwner'] TypeError: 'NoneType' object is not subscriptable
Ah, I've seen a similar issue while working on a newer feature. The GraphQL API can be a bit weird with what it returns (it'll be a 200 but the expected fields might not be present).
Will have a patch in to main and dev when I get some time. If you need this to work in the interim, try adding the following to construct_workflow_cache
in repository.py
:
Old:
for result in yml_results:
owner = result['nameWithOwner']
self.workflow_cache[owner] = list()
if not result['object']:
continue
New:
for result in yml_results:
# If we get any malformed/missing data just skip it and
# Gato will fall back to the contents API for these few cases.
if not result:
continue
if 'nameWithOwner' not in result:
continue
owner = result['nameWithOwner']
# Empty means no yamls, so just skip.
if not result['object']:
self.workflow_cache[owner] = list()
continue
@json-yo can you check out fix/result_error_handling
and run gato? I'm curious if you run into the same issue.
Hi @AdnaneKhan i checked out the new branch fix/result_error_handling
[+] Querying and caching workflow YAML files!
Traceback (most recent call last):
File "/Users/geekfreak/gato/venv/bin/gato", line 8, in <module>
sys.exit(entry())
^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/main.py", line 6, in entry
sys.exit(cli.cli(sys.argv[1:]))
^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/cli/cli.py", line 83, in cli
arguments.func(arguments, subparsers)
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/cli/cli.py", line 242, in enumerate
orgs = [gh_enumeration_runner.enumerate_organization(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/enumerate/enumerate.py", line 184, in enumerate_organization
self.repo_e.construct_workflow_cache(result.json()['data']['nodes'])
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/enumerate/repository.py", line 190, in construct_workflow_cache
owner = result['nameWithOwner']
~~~~~~^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable
(venv) geekfreak@localhost gato % git status
On branch fix/result_error_handling
Your branch is up to date with 'origin/fix/result_error_handling'.
nothing to commit, working tree clean
still not working :( it crashes
Hi @AdnaneKhan i checked out the new branch
fix/result_error_handling
[+] Querying and caching workflow YAML files! Traceback (most recent call last): File "/Users/geekfreak/gato/venv/bin/gato", line 8, in <module> sys.exit(entry()) ^^^^^^^ File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/main.py", line 6, in entry sys.exit(cli.cli(sys.argv[1:])) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/cli/cli.py", line 83, in cli arguments.func(arguments, subparsers) File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/cli/cli.py", line 242, in enumerate orgs = [gh_enumeration_runner.enumerate_organization( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/enumerate/enumerate.py", line 184, in enumerate_organization self.repo_e.construct_workflow_cache(result.json()['data']['nodes']) File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/enumerate/repository.py", line 190, in construct_workflow_cache owner = result['nameWithOwner'] ~~~~~~^^^^^^^^^^^^^^^^^ TypeError: 'NoneType' object is not subscriptable (venv) geekfreak@localhost gato % git status On branch fix/result_error_handling Your branch is up to date with 'origin/fix/result_error_handling'. nothing to commit, working tree clean
still not working :( it crashes
Can you confirm you are running the updated code? Line 190 in the new branch is different.
You might need to clear your virtual environment or install it in editable mode (pip install -e .
)
Hi @AdnaneKhan
Checkout fix/result_error_handling
geekfreak@localhost gato % git checkout fix/result_error_handling
Already on 'fix/result_error_handling'
Your branch is up to date with 'origin/fix/result_error_handling'.
Git Pull
geekfreak@localhost gato % git pull
Already up to date.
activiate python variable
geekfreak@localhost gato % python3 -m venv venv
geekfreak@localhost gato % source venv/bin/activate
pip installing the new changes
(venv) geekfreak@localhost gato % pip install .
Processing /Users/geekfreak/gato
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: colorama in ./venv/lib/python3.11/site-packages (from praetorian-gato==1.6.0) (0.4.6)
Requirement already satisfied: requests in ./venv/lib/python3.11/site-packages (from praetorian-gato==1.6.0) (2.31.0)
Requirement already satisfied: pyyaml in ./venv/lib/python3.11/site-packages (from praetorian-gato==1.6.0) (6.0.1)
Requirement already satisfied: packaging in ./venv/lib/python3.11/site-packages (from praetorian-gato==1.6.0) (23.2)
Requirement already satisfied: cryptography in ./venv/lib/python3.11/site-packages (from praetorian-gato==1.6.0) (41.0.7)
Requirement already satisfied: cffi>=1.12 in ./venv/lib/python3.11/site-packages (from cryptography->praetorian-gato==1.6.0) (1.16.0)
Requirement already satisfied: charset-normalizer<4,>=2 in ./venv/lib/python3.11/site-packages (from requests->praetorian-gato==1.6.0) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./venv/lib/python3.11/site-packages (from requests->praetorian-gato==1.6.0) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./venv/lib/python3.11/site-packages (from requests->praetorian-gato==1.6.0) (2.1.0)
Requirement already satisfied: certifi>=2017.4.17 in ./venv/lib/python3.11/site-packages (from requests->praetorian-gato==1.6.0) (2023.11.17)
Requirement already satisfied: pycparser in ./venv/lib/python3.11/site-packages (from cffi>=1.12->cryptography->praetorian-gato==1.6.0) (2.21)
Building wheels for collected packages: praetorian-gato
Building wheel for praetorian-gato (pyproject.toml) ... done
Created wheel for praetorian-gato: filename=praetorian_gato-1.6.0-py3-none-any.whl size=57426 sha256=2400642f88d5ac715880969635c8892b91451d659f864516fbbf92b6696f3fc7
Stored in directory: /private/var/folders/2j/ywbxjmdx3bbcvr3w6qzpml2w0000gn/T/pip-ephem-wheel-cache-opmwrog1/wheels/b4/94/5d/1b195bbaabbeb3f3bb83a9da998a355b46215bcdbccda43b37
Successfully built praetorian-gato
Installing collected packages: praetorian-gato
Attempting uninstall: praetorian-gato
Found existing installation: praetorian-gato 1.6.0
Uninstalling praetorian-gato-1.6.0:
Successfully uninstalled praetorian-gato-1.6.0
Successfully installed praetorian-gato-1.6.0
[notice] A new release of pip is available: 23.3.1 -> 23.3.2
[notice] To update, run: pip install --upgrade pip
Running Against a target Organization
(venv) geekfreak@localhost gato % gato enum --target paytm
No 'GH_TOKEN' environment variable set! Please enter a GitHub PAT.
......
[+] The GitHub Classic PAT has the following scopes: gist, notifications, read:discussion, read:org, read:packages, read:public_key, read:repo_hook, read:user, repo
[+] Enumerating the paytm organization!
[!] The user has only public access!
Traceback (most recent call last):
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1096, in _validate_conn
conn.connect()
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connection.py", line 642, in connect
sock_and_verified = _ssl_wrap_socket_and_match_hostname(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connection.py", line 782, in _ssl_wrap_socket_and_match_hostname
ssl_sock = ssl_wrap_socket(
^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 470, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 514, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 517, in wrap_socket
return self.sslsocket_class._create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1108, in _create
self.do_handshake()
File "/opt/homebrew/Cellar/python@3.11/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1379, in do_handshake
self._sslobj.do_handshake()
TimeoutError: [Errno 60] Operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 844, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
raise reraise(type(error), error, _stacktrace)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise
raise value
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 469, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.github.com', port=443): Read timed out. (read timeout=None)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/geekfreak/gato/venv/bin/gato", line 8, in <module>
sys.exit(entry())
^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/main.py", line 6, in entry
sys.exit(cli.cli(sys.argv[1:]))
^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/cli/cli.py", line 83, in cli
arguments.func(arguments, subparsers)
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/cli/cli.py", line 242, in enumerate
orgs = [gh_enumeration_runner.enumerate_organization(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/enumerate/enumerate.py", line 168, in enumerate_organization
enum_list = self.org_e.construct_repo_enum_list(organization)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/enumerate/organization.py", line 47, in construct_repo_enum_list
org_private_repos = self.__assemble_repo_list(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/enumerate/organization.py", line 30, in __assemble_repo_list
raw_repos = self.api.check_org_repos(organization, visibility)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/github/api.py", line 550, in check_org_repos
org_repos = self.call_get(f'/orgs/{org}/repos', params=get_params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/gato/github/api.py", line 216, in call_get
api_response = requests.get(request_url, headers=get_header,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/geekfreak/gato/venv/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.github.com', port=443): Read timed out. (read timeout=None)
after that i got the above error
Let me know if i have missed out some steps
Thanks
The original issue with the workflow caching system has been fixed and merged to main. Please let us know if you run into any similar issues here or in a separate issue. Thanks!
I ran gato against a target repo with a 3rd party PAT (i.e. personal one) and it worked fine for enumeration. I then used a PAT for my work (classic PAT like before) GH account and when I tried to enumerate, it throws a NoneType error complaining about it not being subscriptable.
Additional Info:
gato -s e -t XXXXX --output-yaml . --output-json XXXXX.json [+] The authenticated user is: XXXX [!] The token has no scopes! [+] Enumerating the XXXXX organization! [!] The user has only public access! [+] About to enumerate 379 repos within the XXXXX organization! [+] Querying and caching workflow YAML files! Traceback (most recent call last): File "/root/gato/venv/bin/gato", line 8, in
sys.exit(entry())
File "/root/gato/venv/lib/python3.10/site-packages/gato/main.py", line 6, in entry
sys.exit(cli.cli(sys.argv[1:]))
File "/root/gato/venv/lib/python3.10/site-packages/gato/cli/cli.py", line 83, in cli
arguments.func(arguments, subparsers)
File "/root/gato/venv/lib/python3.10/site-packages/gato/cli/cli.py", line 242, in enumerate
orgs = [gh_enumeration_runner.enumerate_organization(
File "/root/gato/venv/lib/python3.10/site-packages/gato/enumerate/enumerate.py", line 184, in enumerate_organization
self.repo_e.construct_workflow_cache(result.json()['data']['nodes'])
File "/root/gato/venv/lib/python3.10/site-packages/gato/enumerate/repository.py", line 190, in construct_workflow_cache
owner = result['nameWithOwner']
TypeError: 'NoneType' object is not subscriptable