glato / emerge

Emerge is a browser-based interactive codebase and dependency visualization tool for many different programming languages. It supports some basic code quality and graph metrics and provides a simple and intuitive way to explore and analyze a codebase by using graph structures.
MIT License
783 stars 46 forks source link

PermissionError: [Errno 13] Permission denied: '/tmp/emerge/source/.git/config.lock' #46

Closed jason-karuza-afresh closed 10 months ago

jason-karuza-afresh commented 11 months ago

Describe the bug Getting a "Permission denied" with .git/config.lock error when trying to generate git-based.

2023-08-04 23:24:50   analysis I ⏩ calculating code metric results for: git metrics
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 941, in _obtain_lock_or_raise
    fd = os.open(lock_file, flags, 0)
PermissionError: [Errno 13] Permission denied: '/tmp/emerge/source/.git/config.lock'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/emerge", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.10/dist-packages/emerge/main.py", line 13, in run
    emerge.start()
  File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 91, in start
    self.start_analyzing()
  File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 112, in start_analyzing
    analyzer.start_analyzing()
  File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 51, in start_analyzing
    self.start_scanning(analysis)
  File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 90, in start_scanning
    self._calculate_code_metric_results(analysis)
  File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 196, in _calculate_code_metric_results
    analysis.calculate_metric(metric)
  File "/usr/local/lib/python3.10/dist-packages/emerge/analysis.py", line 265, in calculate_metric
    metric.calculate_from_results(filtered_results)
  File "/usr/local/lib/python3.10/dist-packages/emerge/metrics/git/git.py", line 86, in calculate_from_results
    self._calculate_git_metrics(results)
  File "/usr/local/lib/python3.10/dist-packages/emerge/metrics/git/git.py", line 113, in _calculate_git_metrics
    for commit in repository.traverse_commits():
  File "/usr/local/lib/python3.10/dist-packages/pydriller/repository.py", line 215, in traverse_commits
    with self._prep_repo(path_repo=path_repo) as git:
  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/pydriller/repository.py", line 186, in _prep_repo
    self.git = Git(local_path_repo, self._conf)
  File "/usr/local/lib/python3.10/dist-packages/pydriller/git.py", line 61, in __init__
    self._open_repository()
  File "/usr/local/lib/python3.10/dist-packages/pydriller/git.py", line 88, in _open_repository
    self._repo.config_writer().set_value("blame", "markUnblamableLines", "true").release()
  File "/usr/local/lib/python3.10/dist-packages/git/repo/base.py", line 596, in config_writer
    return GitConfigParser(self._get_config_path(config_level), read_only=False, repo=self)
  File "/usr/local/lib/python3.10/dist-packages/git/config.py", line 366, in __init__
    self._acquire_lock()
  File "/usr/local/lib/python3.10/dist-packages/git/config.py", line 385, in _acquire_lock
    self._lock._obtain_lock()
  File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 951, in _obtain_lock
    return self._obtain_lock_or_raise()
  File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 944, in _obtain_lock_or_raise
    raise IOError(str(e)) from e
OSError: [Errno 13] Permission denied: '/tmp/emerge/source/.git/config.lock'

Describe your environment

To Reproduce Steps to reproduce the behavior:

  1. Create a config like the above
  2. Run the tool using docker docker run --rm -v /Users/<my_path>/emerge:/tmp/emerge achtelik/emerge:2.0.0 /tmp/emerge/config.yml

Expected behavior Git metrics to be produced

Additional context Running this on a private git repo, but one that I have access to pull etc. If I cd into source and do a git pull it updates the branch.

image
glato commented 11 months ago

@jason-karuza-afresh Thanks for the issue feedback. By briefly looking at this, it seems like this is a file permission issue indeed. Was the file "config.lock" there before you started the analysis with emerge or was it created by emerge (and the python git framework underneath, that is responsible for git mining). From the screenshot you show, no one actually has any permission to read (r), write (w) or execute (x) this file "----------". Looks like this lock also has no size, I guess you might simply delete this lock and try to rerun the analysis?

jason-karuza-afresh commented 11 months ago

Thanks for getting back to me. The config.lock file did not exist prior to running the tool. I'll try deleting and re-running to see if it changes the results, but I wouldn't expect it to.

glato commented 11 months ago

@jason-karuza-afresh another idea: you might simply add some basic read/write permissions with the chmod command (e.g. „chmod 664 config.lock“ in this directory).

jason-karuza-afresh commented 11 months ago

I tried it with chmod 644

image

and it produced a different error.

2023-08-15 20:24:25   analysis I ⏩ calculating code metric results for: git metrics
Traceback (most recent call last):
  File "/usr/local/bin/emerge", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.10/dist-packages/emerge/main.py", line 13, in run
    emerge.start()
  File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 91, in start
    self.start_analyzing()
  File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 112, in start_analyzing
    analyzer.start_analyzing()
  File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 51, in start_analyzing
    self.start_scanning(analysis)
  File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 90, in start_scanning
    self._calculate_code_metric_results(analysis)
  File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 196, in _calculate_code_metric_results
    analysis.calculate_metric(metric)
  File "/usr/local/lib/python3.10/dist-packages/emerge/analysis.py", line 265, in calculate_metric
    metric.calculate_from_results(filtered_results)
  File "/usr/local/lib/python3.10/dist-packages/emerge/metrics/git/git.py", line 86, in calculate_from_results
    self._calculate_git_metrics(results)
  File "/usr/local/lib/python3.10/dist-packages/emerge/metrics/git/git.py", line 113, in _calculate_git_metrics
    for commit in repository.traverse_commits():
  File "/usr/local/lib/python3.10/dist-packages/pydriller/repository.py", line 215, in traverse_commits
    with self._prep_repo(path_repo=path_repo) as git:
  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/pydriller/repository.py", line 186, in _prep_repo
    self.git = Git(local_path_repo, self._conf)
  File "/usr/local/lib/python3.10/dist-packages/pydriller/git.py", line 61, in __init__
    self._open_repository()
  File "/usr/local/lib/python3.10/dist-packages/pydriller/git.py", line 88, in _open_repository
    self._repo.config_writer().set_value("blame", "markUnblamableLines", "true").release()
  File "/usr/local/lib/python3.10/dist-packages/git/repo/base.py", line 596, in config_writer
    return GitConfigParser(self._get_config_path(config_level), read_only=False, repo=self)
  File "/usr/local/lib/python3.10/dist-packages/git/config.py", line 366, in __init__
    self._acquire_lock()
  File "/usr/local/lib/python3.10/dist-packages/git/config.py", line 385, in _acquire_lock
    self._lock._obtain_lock()
  File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 951, in _obtain_lock
    return self._obtain_lock_or_raise()
  File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 932, in _obtain_lock_or_raise
    raise IOError(
OSError: Lock for file '/tmp/emerge/source/.git/config' did already exist, delete '/tmp/emerge/source/.git/config.lock' in case the lock is illegal

I then tried deleting it and it was regenerated with the same error.

image
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 941, in _obtain_lock_or_raise
    fd = os.open(lock_file, flags, 0)
PermissionError: [Errno 13] Permission denied: '/tmp/emerge/source/.git/config.lock'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/emerge", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.10/dist-packages/emerge/main.py", line 13, in run
    emerge.start()
  File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 91, in start
    self.start_analyzing()
  File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 112, in start_analyzing
    analyzer.start_analyzing()
  File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 51, in start_analyzing
    self.start_scanning(analysis)
  File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 90, in start_scanning
    self._calculate_code_metric_results(analysis)
  File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 196, in _calculate_code_metric_results
    analysis.calculate_metric(metric)
  File "/usr/local/lib/python3.10/dist-packages/emerge/analysis.py", line 265, in calculate_metric
    metric.calculate_from_results(filtered_results)
  File "/usr/local/lib/python3.10/dist-packages/emerge/metrics/git/git.py", line 86, in calculate_from_results
    self._calculate_git_metrics(results)
  File "/usr/local/lib/python3.10/dist-packages/emerge/metrics/git/git.py", line 113, in _calculate_git_metrics
    for commit in repository.traverse_commits():
  File "/usr/local/lib/python3.10/dist-packages/pydriller/repository.py", line 215, in traverse_commits
    with self._prep_repo(path_repo=path_repo) as git:
  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/pydriller/repository.py", line 186, in _prep_repo
    self.git = Git(local_path_repo, self._conf)
  File "/usr/local/lib/python3.10/dist-packages/pydriller/git.py", line 61, in __init__
    self._open_repository()
  File "/usr/local/lib/python3.10/dist-packages/pydriller/git.py", line 88, in _open_repository
    self._repo.config_writer().set_value("blame", "markUnblamableLines", "true").release()
  File "/usr/local/lib/python3.10/dist-packages/git/repo/base.py", line 596, in config_writer
    return GitConfigParser(self._get_config_path(config_level), read_only=False, repo=self)
  File "/usr/local/lib/python3.10/dist-packages/git/config.py", line 366, in __init__
    self._acquire_lock()
  File "/usr/local/lib/python3.10/dist-packages/git/config.py", line 385, in _acquire_lock
    self._lock._obtain_lock()
  File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 951, in _obtain_lock
    return self._obtain_lock_or_raise()
  File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 944, in _obtain_lock_or_raise
    raise IOError(str(e)) from e
OSError: [Errno 13] Permission denied: '/tmp/emerge/source/.git/config.lock'

Other ideas?

jason-karuza-afresh commented 11 months ago

Following up in hopes there is something that I am obviously doing wrong. If you generate code metrics do you have the same issue at all? I did a clean clone from git and still have the same issue.

ornithos commented 10 months ago

Can reproduce @jason-karuza-afresh :+1: . For what it's worth I first ran into the error at the point of export (I wasn't running git metrics): i.e. from running docker run --rm -v /home/sandbox/emerge_input:/tmp/emerge achtelik/emerge:2.0.0 /tmp/emerge/config.yml I obtained:

...
2023-08-30 11:19:34   analysis I ⏩ calculating graph metric results for: louvain modularity metric
2023-08-30 11:19:35   analysis I ✅ done calculating graph metric results
Traceback (most recent call last):
  File "/usr/local/bin/emerge", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.10/dist-packages/emerge/main.py", line 13, in run
    emerge.start()
  File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 91, in start
    self.start_analyzing()
  File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 112, in start_analyzing
    analyzer.start_analyzing()
  File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 51, in start_analyzing
    self.start_scanning(analysis)
  File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 103, in start_scanning
    analysis.export()
  File "/usr/local/lib/python3.10/dist-packages/emerge/analysis.py", line 298, in export
    GraphExporter.export_graph_as_graphml(representation.digraph, representation.graph_type.name.lower(), self.export_directory)
  File "/usr/local/lib/python3.10/dist-packages/emerge/export.py", line 34, in export_graph_as_graphml
    nx.write_graphml(graph, export_dir + '/' + 'emerge-' + export_name + '.graphml')
  File "/usr/local/lib/python3.10/dist-packages/networkx/utils/decorators.py", line 766, in func
    return argmap._lazy_compile(__wrapper)(*args, **kwargs)
  File "<class 'networkx.utils.decorators.argmap'> compilation 5", line 3, in argmap_write_graphml_lxml_1
  File "/usr/local/lib/python3.10/dist-packages/networkx/utils/decorators.py", line 189, in _open_file
    fobj = _dispatch_dict[ext](path, mode=mode)
PermissionError: [Errno 13] Permission denied: '/tmp/emerge/export/emerge-file_result_dependency_graph.graphml'

I've run into my fair share of Docker-related permission errors before, all of which have been a pain to solve. It seems likely that in your case when GitPython / PyDriller initialises it temporarily locks the config file for Git, and encounters the permissions error much earlier.

What I can tell you is that installing and running locally following the instructions here worked like a charm. Maybe worth avoiding Docker altogether for now?

jason-karuza-afresh commented 10 months ago

Thanks @ornithos . I initially started with trying to run the tool without docker, too. Kept having issues with the pyenv or pip setup. I just tried again and got an error (I am using a M1 Macbook):

jasonkaruza @ ~/emerge - [] $ pip install emerge-viz
Defaulting to user installation because normal site-packages is not writeable
Collecting emerge-viz
  Obtaining dependency information for emerge-viz from https://files.pythonhosted.org/packages/86/7c/c30e511b58c670277383b0f60f56d8e604cd18f94b44d1a7d03bbe000562/emerge_viz-2.0.0-py3-none-any.whl.metadata
  Using cached emerge_viz-2.0.0-py3-none-any.whl.metadata (33 kB)
Requirement already satisfied: wheel in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages (from emerge-viz) (0.37.0)
Collecting autopep8 (from emerge-viz)
  Obtaining dependency information for autopep8 from https://files.pythonhosted.org/packages/d8/f2/e63c9f9c485cd90df8e4e7ae90fa3be2469c9641888558c7b45fa98a76f8/autopep8-2.0.4-py2.py3-none-any.whl.metadata
  Downloading autopep8-2.0.4-py2.py3-none-any.whl.metadata (17 kB)
Collecting coloredlogs (from emerge-viz)
  Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
Collecting interrogate (from emerge-viz)
  Using cached interrogate-1.5.0-py3-none-any.whl (45 kB)
Collecting networkx (from emerge-viz)
  Using cached networkx-3.1-py3-none-any.whl (2.1 MB)
Collecting scikit-learn (from emerge-viz)
  Using cached scikit-learn-1.3.0.tar.gz (7.5 MB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [83 lines of output]
      Ignoring numpy: markers 'python_version == "3.10" and platform_system == "Windows" and platform_python_implementation != "PyPy"' don't match your environment
      Collecting setuptools
        Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/4f/ab/0bcfebdfc3bfa8554b2b2c97a555569c4c1ebc74ea288741ea8326c51906/setuptools-68.1.2-py3-none-any.whl.metadata
        Downloading setuptools-68.1.2-py3-none-any.whl.metadata (6.2 kB)
      Collecting wheel
        Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/b8/8b/31273bf66016be6ad22bb7345c37ff350276cfd46e389a0c2ac5da9d9073/wheel-0.41.2-py3-none-any.whl.metadata
        Downloading wheel-0.41.2-py3-none-any.whl.metadata (2.2 kB)
      Collecting Cython>=0.29.33
        Obtaining dependency information for Cython>=0.29.33 from https://files.pythonhosted.org/packages/03/e9/9cc0c4f0d8a566089d096254cd25168a0db02dd047863a7f995d8d3eefa7/Cython-3.0.2-py2.py3-none-any.whl.metadata
        Downloading Cython-3.0.2-py2.py3-none-any.whl.metadata (3.1 kB)
      Collecting oldest-supported-numpy
        Obtaining dependency information for oldest-supported-numpy from https://files.pythonhosted.org/packages/94/9a/756fef9346e5ca2289cb70d73990b4c9f25446a885c1186cfb93a85e7da0/oldest_supported_numpy-2023.8.3-py3-none-any.whl.metadata
        Using cached oldest_supported_numpy-2023.8.3-py3-none-any.whl.metadata (9.5 kB)
      Collecting scipy>=1.5.0
        Downloading scipy-1.11.2.tar.gz (56.0 MB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.0/56.0 MB 10.5 MB/s eta 0:00:00
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'error'
        error: subprocess-exited-with-error

        × Preparing metadata (pyproject.toml) did not run successfully.
        │ exit code: 1
        ╰─> [44 lines of output]
            + meson setup /private/var/folders/vd/lmyqmyns157br_h68lm7ynqc0000gn/T/pip-install-510xa43m/scipy_3971be73e6eb410c8ea72988cb8bac76 /private/var/folders/vd/lmyqmyns157br_h68lm7ynqc0000gn/T/pip-install-510xa43m/scipy_3971be73e6eb410c8ea72988cb8bac76/.mesonpy-wk3wtgxl/build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/private/var/folders/vd/lmyqmyns157br_h68lm7ynqc0000gn/T/pip-install-510xa43m/scipy_3971be73e6eb410c8ea72988cb8bac76/.mesonpy-wk3wtgxl/build/meson-python-native-file.ini
            The Meson build system
            Version: 1.2.1
            Source dir: /private/var/folders/vd/lmyqmyns157br_h68lm7ynqc0000gn/T/pip-install-510xa43m/scipy_3971be73e6eb410c8ea72988cb8bac76
            Build dir: /private/var/folders/vd/lmyqmyns157br_h68lm7ynqc0000gn/T/pip-install-510xa43m/scipy_3971be73e6eb410c8ea72988cb8bac76/.mesonpy-wk3wtgxl/build
            Build type: native build
            Project name: SciPy
            Project version: 1.11.2
            C compiler for the host machine: cc (clang 14.0.3 "Apple clang version 14.0.3 (clang-1403.0.22.14.1)")
            C linker for the host machine: cc ld64 857.1
            C++ compiler for the host machine: c++ (clang 14.0.3 "Apple clang version 14.0.3 (clang-1403.0.22.14.1)")
            C++ linker for the host machine: c++ ld64 857.1
            Cython compiler for the host machine: cython (cython 0.29.36)
            Host machine cpu family: aarch64
            Host machine cpu: aarch64
            Program python found: YES (/Library/Developer/CommandLineTools/usr/bin/python3)
            Did not find pkg-config by name 'pkg-config'
            Found Pkg-config: NO
            Run-time dependency python found: YES 3.9
            Program cython found: YES (/private/var/folders/vd/lmyqmyns157br_h68lm7ynqc0000gn/T/pip-build-env-4282tikc/overlay/bin/cython)
            Compiler for C supports arguments -Wno-unused-but-set-variable: YES
            Compiler for C supports arguments -Wno-unused-function: YES
            Compiler for C supports arguments -Wno-conversion: YES
            Compiler for C supports arguments -Wno-misleading-indentation: YES
            Library m found: YES

            ../../meson.build:82:0: ERROR: Unknown compiler(s): [['gfortran'], ['flang'], ['nvfortran'], ['pgfortran'], ['ifort'], ['ifx'], ['g95']]
            The following exception(s) were encountered:
            Running `gfortran --version` gave "[Errno 2] No such file or directory: 'gfortran'"
            Running `gfortran -V` gave "[Errno 2] No such file or directory: 'gfortran'"
            Running `flang --version` gave "[Errno 2] No such file or directory: 'flang'"
            Running `flang -V` gave "[Errno 2] No such file or directory: 'flang'"
            Running `nvfortran --version` gave "[Errno 2] No such file or directory: 'nvfortran'"
            Running `nvfortran -V` gave "[Errno 2] No such file or directory: 'nvfortran'"
            Running `pgfortran --version` gave "[Errno 2] No such file or directory: 'pgfortran'"
            Running `pgfortran -V` gave "[Errno 2] No such file or directory: 'pgfortran'"
            Running `ifort --version` gave "[Errno 2] No such file or directory: 'ifort'"
            Running `ifort -V` gave "[Errno 2] No such file or directory: 'ifort'"
            Running `ifx --version` gave "[Errno 2] No such file or directory: 'ifx'"
            Running `ifx -V` gave "[Errno 2] No such file or directory: 'ifx'"
            Running `g95 --version` gave "[Errno 2] No such file or directory: 'g95'"
            Running `g95 -V` gave "[Errno 2] No such file or directory: 'g95'"

            A full log can be found at /private/var/folders/vd/lmyqmyns157br_h68lm7ynqc0000gn/T/pip-install-510xa43m/scipy_3971be73e6eb410c8ea72988cb8bac76/.mesonpy-wk3wtgxl/build/meson-logs/meson-log.txt
            [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: metadata-generation-failed

      × Encountered error while generating package metadata.
      ╰─> See above for output.

      note: This is an issue with the package mentioned above, not pip.
      hint: See above for details.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
ornithos commented 10 months ago

Perhaps you're missing a fortran compiler. E.g. https://stackoverflow.com/a/73421376. It should be sufficient to install GCC e.g. brew install gcc; if it's still not working, you could try using conda instead of pip which historically has handled this stuff better. If all else fails you could try the user install instructions inside an interactive linux Docker container.

jason-karuza-afresh commented 10 months ago

Installing gcc didn't help, but installing openblas and pkg-config via brew (and adding the various env flag paths for openblas) did.

Short summary seems to be: don't generate source code analysis in docker mode.

Thanks y'all!