Closed jason-karuza-afresh closed 10 months ago
@jason-karuza-afresh Thanks for the issue feedback. By briefly looking at this, it seems like this is a file permission issue indeed. Was the file "config.lock" there before you started the analysis with emerge or was it created by emerge (and the python git framework underneath, that is responsible for git mining). From the screenshot you show, no one actually has any permission to read (r), write (w) or execute (x) this file "----------". Looks like this lock also has no size, I guess you might simply delete this lock and try to rerun the analysis?
Thanks for getting back to me. The config.lock
file did not exist prior to running the tool. I'll try deleting and re-running to see if it changes the results, but I wouldn't expect it to.
@jason-karuza-afresh another idea: you might simply add some basic read/write permissions with the chmod command (e.g. „chmod 664 config.lock“ in this directory).
I tried it with chmod 644
and it produced a different error.
2023-08-15 20:24:25 analysis I ⏩ calculating code metric results for: git metrics
Traceback (most recent call last):
File "/usr/local/bin/emerge", line 8, in <module>
sys.exit(run())
File "/usr/local/lib/python3.10/dist-packages/emerge/main.py", line 13, in run
emerge.start()
File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 91, in start
self.start_analyzing()
File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 112, in start_analyzing
analyzer.start_analyzing()
File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 51, in start_analyzing
self.start_scanning(analysis)
File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 90, in start_scanning
self._calculate_code_metric_results(analysis)
File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 196, in _calculate_code_metric_results
analysis.calculate_metric(metric)
File "/usr/local/lib/python3.10/dist-packages/emerge/analysis.py", line 265, in calculate_metric
metric.calculate_from_results(filtered_results)
File "/usr/local/lib/python3.10/dist-packages/emerge/metrics/git/git.py", line 86, in calculate_from_results
self._calculate_git_metrics(results)
File "/usr/local/lib/python3.10/dist-packages/emerge/metrics/git/git.py", line 113, in _calculate_git_metrics
for commit in repository.traverse_commits():
File "/usr/local/lib/python3.10/dist-packages/pydriller/repository.py", line 215, in traverse_commits
with self._prep_repo(path_repo=path_repo) as git:
File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.10/dist-packages/pydriller/repository.py", line 186, in _prep_repo
self.git = Git(local_path_repo, self._conf)
File "/usr/local/lib/python3.10/dist-packages/pydriller/git.py", line 61, in __init__
self._open_repository()
File "/usr/local/lib/python3.10/dist-packages/pydriller/git.py", line 88, in _open_repository
self._repo.config_writer().set_value("blame", "markUnblamableLines", "true").release()
File "/usr/local/lib/python3.10/dist-packages/git/repo/base.py", line 596, in config_writer
return GitConfigParser(self._get_config_path(config_level), read_only=False, repo=self)
File "/usr/local/lib/python3.10/dist-packages/git/config.py", line 366, in __init__
self._acquire_lock()
File "/usr/local/lib/python3.10/dist-packages/git/config.py", line 385, in _acquire_lock
self._lock._obtain_lock()
File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 951, in _obtain_lock
return self._obtain_lock_or_raise()
File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 932, in _obtain_lock_or_raise
raise IOError(
OSError: Lock for file '/tmp/emerge/source/.git/config' did already exist, delete '/tmp/emerge/source/.git/config.lock' in case the lock is illegal
I then tried deleting it and it was regenerated with the same error.
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 941, in _obtain_lock_or_raise
fd = os.open(lock_file, flags, 0)
PermissionError: [Errno 13] Permission denied: '/tmp/emerge/source/.git/config.lock'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/emerge", line 8, in <module>
sys.exit(run())
File "/usr/local/lib/python3.10/dist-packages/emerge/main.py", line 13, in run
emerge.start()
File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 91, in start
self.start_analyzing()
File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 112, in start_analyzing
analyzer.start_analyzing()
File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 51, in start_analyzing
self.start_scanning(analysis)
File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 90, in start_scanning
self._calculate_code_metric_results(analysis)
File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 196, in _calculate_code_metric_results
analysis.calculate_metric(metric)
File "/usr/local/lib/python3.10/dist-packages/emerge/analysis.py", line 265, in calculate_metric
metric.calculate_from_results(filtered_results)
File "/usr/local/lib/python3.10/dist-packages/emerge/metrics/git/git.py", line 86, in calculate_from_results
self._calculate_git_metrics(results)
File "/usr/local/lib/python3.10/dist-packages/emerge/metrics/git/git.py", line 113, in _calculate_git_metrics
for commit in repository.traverse_commits():
File "/usr/local/lib/python3.10/dist-packages/pydriller/repository.py", line 215, in traverse_commits
with self._prep_repo(path_repo=path_repo) as git:
File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/usr/local/lib/python3.10/dist-packages/pydriller/repository.py", line 186, in _prep_repo
self.git = Git(local_path_repo, self._conf)
File "/usr/local/lib/python3.10/dist-packages/pydriller/git.py", line 61, in __init__
self._open_repository()
File "/usr/local/lib/python3.10/dist-packages/pydriller/git.py", line 88, in _open_repository
self._repo.config_writer().set_value("blame", "markUnblamableLines", "true").release()
File "/usr/local/lib/python3.10/dist-packages/git/repo/base.py", line 596, in config_writer
return GitConfigParser(self._get_config_path(config_level), read_only=False, repo=self)
File "/usr/local/lib/python3.10/dist-packages/git/config.py", line 366, in __init__
self._acquire_lock()
File "/usr/local/lib/python3.10/dist-packages/git/config.py", line 385, in _acquire_lock
self._lock._obtain_lock()
File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 951, in _obtain_lock
return self._obtain_lock_or_raise()
File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 944, in _obtain_lock_or_raise
raise IOError(str(e)) from e
OSError: [Errno 13] Permission denied: '/tmp/emerge/source/.git/config.lock'
Other ideas?
Following up in hopes there is something that I am obviously doing wrong. If you generate code metrics do you have the same issue at all? I did a clean clone from git and still have the same issue.
Can reproduce @jason-karuza-afresh :+1: . For what it's worth I first ran into the error at the point of export (I wasn't running git
metrics):
i.e. from running docker run --rm -v /home/sandbox/emerge_input:/tmp/emerge achtelik/emerge:2.0.0 /tmp/emerge/config.yml
I obtained:
...
2023-08-30 11:19:34 analysis I ⏩ calculating graph metric results for: louvain modularity metric
2023-08-30 11:19:35 analysis I ✅ done calculating graph metric results
Traceback (most recent call last):
File "/usr/local/bin/emerge", line 8, in <module>
sys.exit(run())
File "/usr/local/lib/python3.10/dist-packages/emerge/main.py", line 13, in run
emerge.start()
File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 91, in start
self.start_analyzing()
File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 112, in start_analyzing
analyzer.start_analyzing()
File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 51, in start_analyzing
self.start_scanning(analysis)
File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 103, in start_scanning
analysis.export()
File "/usr/local/lib/python3.10/dist-packages/emerge/analysis.py", line 298, in export
GraphExporter.export_graph_as_graphml(representation.digraph, representation.graph_type.name.lower(), self.export_directory)
File "/usr/local/lib/python3.10/dist-packages/emerge/export.py", line 34, in export_graph_as_graphml
nx.write_graphml(graph, export_dir + '/' + 'emerge-' + export_name + '.graphml')
File "/usr/local/lib/python3.10/dist-packages/networkx/utils/decorators.py", line 766, in func
return argmap._lazy_compile(__wrapper)(*args, **kwargs)
File "<class 'networkx.utils.decorators.argmap'> compilation 5", line 3, in argmap_write_graphml_lxml_1
File "/usr/local/lib/python3.10/dist-packages/networkx/utils/decorators.py", line 189, in _open_file
fobj = _dispatch_dict[ext](path, mode=mode)
PermissionError: [Errno 13] Permission denied: '/tmp/emerge/export/emerge-file_result_dependency_graph.graphml'
I've run into my fair share of Docker-related permission errors before, all of which have been a pain to solve. It seems likely that in your case when GitPython / PyDriller initialises it temporarily locks the config file for Git, and encounters the permissions error much earlier.
What I can tell you is that installing and running locally following the instructions here worked like a charm. Maybe worth avoiding Docker altogether for now?
Thanks @ornithos . I initially started with trying to run the tool without docker, too. Kept having issues with the pyenv or pip setup. I just tried again and got an error (I am using a M1 Macbook):
jasonkaruza @ ~/emerge - [] $ pip install emerge-viz
Defaulting to user installation because normal site-packages is not writeable
Collecting emerge-viz
Obtaining dependency information for emerge-viz from https://files.pythonhosted.org/packages/86/7c/c30e511b58c670277383b0f60f56d8e604cd18f94b44d1a7d03bbe000562/emerge_viz-2.0.0-py3-none-any.whl.metadata
Using cached emerge_viz-2.0.0-py3-none-any.whl.metadata (33 kB)
Requirement already satisfied: wheel in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages (from emerge-viz) (0.37.0)
Collecting autopep8 (from emerge-viz)
Obtaining dependency information for autopep8 from https://files.pythonhosted.org/packages/d8/f2/e63c9f9c485cd90df8e4e7ae90fa3be2469c9641888558c7b45fa98a76f8/autopep8-2.0.4-py2.py3-none-any.whl.metadata
Downloading autopep8-2.0.4-py2.py3-none-any.whl.metadata (17 kB)
Collecting coloredlogs (from emerge-viz)
Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
Collecting interrogate (from emerge-viz)
Using cached interrogate-1.5.0-py3-none-any.whl (45 kB)
Collecting networkx (from emerge-viz)
Using cached networkx-3.1-py3-none-any.whl (2.1 MB)
Collecting scikit-learn (from emerge-viz)
Using cached scikit-learn-1.3.0.tar.gz (7.5 MB)
Installing build dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [83 lines of output]
Ignoring numpy: markers 'python_version == "3.10" and platform_system == "Windows" and platform_python_implementation != "PyPy"' don't match your environment
Collecting setuptools
Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/4f/ab/0bcfebdfc3bfa8554b2b2c97a555569c4c1ebc74ea288741ea8326c51906/setuptools-68.1.2-py3-none-any.whl.metadata
Downloading setuptools-68.1.2-py3-none-any.whl.metadata (6.2 kB)
Collecting wheel
Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/b8/8b/31273bf66016be6ad22bb7345c37ff350276cfd46e389a0c2ac5da9d9073/wheel-0.41.2-py3-none-any.whl.metadata
Downloading wheel-0.41.2-py3-none-any.whl.metadata (2.2 kB)
Collecting Cython>=0.29.33
Obtaining dependency information for Cython>=0.29.33 from https://files.pythonhosted.org/packages/03/e9/9cc0c4f0d8a566089d096254cd25168a0db02dd047863a7f995d8d3eefa7/Cython-3.0.2-py2.py3-none-any.whl.metadata
Downloading Cython-3.0.2-py2.py3-none-any.whl.metadata (3.1 kB)
Collecting oldest-supported-numpy
Obtaining dependency information for oldest-supported-numpy from https://files.pythonhosted.org/packages/94/9a/756fef9346e5ca2289cb70d73990b4c9f25446a885c1186cfb93a85e7da0/oldest_supported_numpy-2023.8.3-py3-none-any.whl.metadata
Using cached oldest_supported_numpy-2023.8.3-py3-none-any.whl.metadata (9.5 kB)
Collecting scipy>=1.5.0
Downloading scipy-1.11.2.tar.gz (56.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.0/56.0 MB 10.5 MB/s eta 0:00:00
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Installing backend dependencies: started
Installing backend dependencies: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'error'
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [44 lines of output]
+ meson setup /private/var/folders/vd/lmyqmyns157br_h68lm7ynqc0000gn/T/pip-install-510xa43m/scipy_3971be73e6eb410c8ea72988cb8bac76 /private/var/folders/vd/lmyqmyns157br_h68lm7ynqc0000gn/T/pip-install-510xa43m/scipy_3971be73e6eb410c8ea72988cb8bac76/.mesonpy-wk3wtgxl/build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/private/var/folders/vd/lmyqmyns157br_h68lm7ynqc0000gn/T/pip-install-510xa43m/scipy_3971be73e6eb410c8ea72988cb8bac76/.mesonpy-wk3wtgxl/build/meson-python-native-file.ini
The Meson build system
Version: 1.2.1
Source dir: /private/var/folders/vd/lmyqmyns157br_h68lm7ynqc0000gn/T/pip-install-510xa43m/scipy_3971be73e6eb410c8ea72988cb8bac76
Build dir: /private/var/folders/vd/lmyqmyns157br_h68lm7ynqc0000gn/T/pip-install-510xa43m/scipy_3971be73e6eb410c8ea72988cb8bac76/.mesonpy-wk3wtgxl/build
Build type: native build
Project name: SciPy
Project version: 1.11.2
C compiler for the host machine: cc (clang 14.0.3 "Apple clang version 14.0.3 (clang-1403.0.22.14.1)")
C linker for the host machine: cc ld64 857.1
C++ compiler for the host machine: c++ (clang 14.0.3 "Apple clang version 14.0.3 (clang-1403.0.22.14.1)")
C++ linker for the host machine: c++ ld64 857.1
Cython compiler for the host machine: cython (cython 0.29.36)
Host machine cpu family: aarch64
Host machine cpu: aarch64
Program python found: YES (/Library/Developer/CommandLineTools/usr/bin/python3)
Did not find pkg-config by name 'pkg-config'
Found Pkg-config: NO
Run-time dependency python found: YES 3.9
Program cython found: YES (/private/var/folders/vd/lmyqmyns157br_h68lm7ynqc0000gn/T/pip-build-env-4282tikc/overlay/bin/cython)
Compiler for C supports arguments -Wno-unused-but-set-variable: YES
Compiler for C supports arguments -Wno-unused-function: YES
Compiler for C supports arguments -Wno-conversion: YES
Compiler for C supports arguments -Wno-misleading-indentation: YES
Library m found: YES
../../meson.build:82:0: ERROR: Unknown compiler(s): [['gfortran'], ['flang'], ['nvfortran'], ['pgfortran'], ['ifort'], ['ifx'], ['g95']]
The following exception(s) were encountered:
Running `gfortran --version` gave "[Errno 2] No such file or directory: 'gfortran'"
Running `gfortran -V` gave "[Errno 2] No such file or directory: 'gfortran'"
Running `flang --version` gave "[Errno 2] No such file or directory: 'flang'"
Running `flang -V` gave "[Errno 2] No such file or directory: 'flang'"
Running `nvfortran --version` gave "[Errno 2] No such file or directory: 'nvfortran'"
Running `nvfortran -V` gave "[Errno 2] No such file or directory: 'nvfortran'"
Running `pgfortran --version` gave "[Errno 2] No such file or directory: 'pgfortran'"
Running `pgfortran -V` gave "[Errno 2] No such file or directory: 'pgfortran'"
Running `ifort --version` gave "[Errno 2] No such file or directory: 'ifort'"
Running `ifort -V` gave "[Errno 2] No such file or directory: 'ifort'"
Running `ifx --version` gave "[Errno 2] No such file or directory: 'ifx'"
Running `ifx -V` gave "[Errno 2] No such file or directory: 'ifx'"
Running `g95 --version` gave "[Errno 2] No such file or directory: 'g95'"
Running `g95 -V` gave "[Errno 2] No such file or directory: 'g95'"
A full log can be found at /private/var/folders/vd/lmyqmyns157br_h68lm7ynqc0000gn/T/pip-install-510xa43m/scipy_3971be73e6eb410c8ea72988cb8bac76/.mesonpy-wk3wtgxl/build/meson-logs/meson-log.txt
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Perhaps you're missing a fortran
compiler. E.g. https://stackoverflow.com/a/73421376. It should be sufficient to install GCC e.g. brew install gcc
; if it's still not working, you could try using conda
instead of pip
which historically has handled this stuff better. If all else fails you could try the user install instructions inside an interactive linux Docker container.
Installing gcc didn't help, but installing openblas and pkg-config via brew (and adding the various env flag paths for openblas) did.
Short summary seems to be: don't generate source code analysis in docker mode.
Thanks y'all!
Describe the bug Getting a "Permission denied" with
.git/config.lock
error when trying to generate git-based.Describe your environment
macos
3.9
Not relevant
no
To Reproduce Steps to reproduce the behavior:
docker run --rm -v /Users/<my_path>/emerge:/tmp/emerge achtelik/emerge:2.0.0 /tmp/emerge/config.yml
Expected behavior Git metrics to be produced
Additional context Running this on a private git repo, but one that I have access to pull etc. If I
cd
intosource
and do agit pull
it updates the branch.