search

CERTCC / labyrinth

Come inside, and have a nice cup of tea.
Other
96 stars 35 forks source link

repo_deep_dive failing with permission error (intermittent) #19

Open ahouseholder opened 1 year ago

ahouseholder commented 1 year ago

See for example https://github.com/CERTCC/labyrinth/actions/runs/5527573575/job/14968312185 log snippet follows

2023-07-12T05:42:25.0531884Z ##[group]Run repo_deep_dive --verbose --mod 3 --divisor 10 --results_dir results/2023/07/11 --max_age 7200
2023-07-12T05:42:25.0532364Z repo_deep_dive --verbose --mod 3 --divisor 10 --results_dir results/2023/07/11 --max_age 7200
2023-07-12T05:42:25.0634838Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-07-12T05:42:25.0635111Z env:
2023-07-12T05:42:25.0635399Z   pythonLocation: /opt/hostedtoolcache/Python/3.9.17/x64
2023-07-12T05:42:25.0635778Z   PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.9.17/x64/lib/pkgconfig
2023-07-12T05:42:25.0636381Z   Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.17/x64
2023-07-12T05:42:25.0636726Z   Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.17/x64
2023-07-12T05:42:25.0637048Z   Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.9.17/x64
2023-07-12T05:42:25.0637388Z   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.17/x64/lib
2023-07-12T05:42:25.0637974Z   GH_TOKEN: ***
2023-07-12T05:42:25.0638187Z ##[endgroup]
2023-07-12T05:42:25.7345778Z INFO root - log level: INFO
2023-07-12T05:42:25.7353976Z INFO labyrinth.repo_processor - Reading 1 search result summaries
2023-07-12T05:42:25.7835639Z INFO labyrinth.repo_processor - Found 16 search results to process
2023-07-12T05:42:25.7888462Z INFO labyrinth.repo_processor - Cloning https://github.com/bha-vin/HTB-Beep.git 1 of 16
2023-07-12T05:42:25.9948417Z INFO labyrinth.repo_processor - Cloning https://github.com/codingcore12/SILENT-DOC-EXPLOIT-CLEAN-v5.git 2 of 16
2023-07-12T05:42:26.1897741Z INFO labyrinth.repo_processor - Cloning https://github.com/gcarrilao/hook.git 3 of 16
2023-07-12T05:42:26.3880162Z Traceback (most recent call last):
2023-07-12T05:42:26.3887717Z   File "/opt/hostedtoolcache/Python/3.9.17/x64/bin/repo_deep_dive", line 57, in <module>
2023-07-12T05:42:26.3888708Z     process_modulo(args.results_dir, args.mod, args.divisor)
2023-07-12T05:42:26.3889799Z   File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/labyrinth/repo_processor.py", line 276, in process_modulo
2023-07-12T05:42:26.3890321Z     df = scan_repos(top_dir, mod, divisor)
2023-07-12T05:42:26.3891004Z   File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/labyrinth/repo_processor.py", line 223, in scan_repos
2023-07-12T05:42:26.3891563Z     results = df.apply(process_row, axis=1).to_list()
2023-07-12T05:42:26.3892230Z   File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pandas/core/frame.py", line 9423, in apply
2023-07-12T05:42:26.3899802Z     return op.apply().__finalize__(self, method="apply")
2023-07-12T05:42:26.3900537Z   File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pandas/core/apply.py", line 678, in apply
2023-07-12T05:42:26.3901242Z     return self.apply_standard()
2023-07-12T05:42:26.3901838Z   File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pandas/core/apply.py", line 798, in apply_standard
2023-07-12T05:42:26.3903187Z     results, res_index = self.apply_series_generator()
2023-07-12T05:42:26.3904231Z   File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/pandas/core/apply.py", line 814, in apply_series_generator
2023-07-12T05:42:26.3905168Z     results[i] = self.f(v)
2023-07-12T05:42:26.3905773Z   File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/labyrinth/repo_processor.py", line 148, in process_row
2023-07-12T05:42:26.3906211Z     _df = process_git_url(clone_url, workdir)
2023-07-12T05:42:26.3906927Z   File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/labyrinth/repo_processor.py", line 49, in process_git_url
2023-07-12T05:42:26.3907353Z     df = process_dir(workdir, workdir)
2023-07-12T05:42:26.3908020Z   File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/labyrinth/file_processor.py", line 94, in process_dir
2023-07-12T05:42:26.3908443Z     _df = process_file(fpath, workdir)
2023-07-12T05:42:26.3909108Z   File "/opt/hostedtoolcache/Python/3.9.17/x64/lib/python3.9/site-packages/labyrinth/file_processor.py", line 28, in process_file
2023-07-12T05:42:26.3909691Z     with open(fpath, "r", encoding="ISO-8859-1") as fp:
2023-07-12T05:42:26.3910183Z PermissionError: [Errno 13] Permission denied: '/tmp/git-clone-svldfy0g/link'
2023-07-12T05:42:26.4827113Z ##[error]Process completed with exit code 1.