search

iterative / dvc

🦉 ML Experiments and Data Management with Git
https://dvc.org
Apache License 2.0
13.6k stars 1.18k forks source link

git hook: 'import dvc' fails #1754

Closed ei-grad closed 5 years ago

ei-grad commented 5 years ago

The import dvc fails on _update_version_file if it is run from the pre-commit git hook. The _update_version_file() function is called if dvc is installed in editable mode (with pip install -e) and during the python -m tests run.

Steps to reproduce

  1. checkout the branch with fixed TestInstall testcase - https://github.com/ei-grad/dvc/tree/test-install
  2. run python -m tests tests/test_install.py.

Traceback

git.exc.HookExecutionError: Hook('/tmp/dvc-test.6040.g8_dftql.64fb4aec-ac74-4af1-9a3e-eec62c6c822a/.git/hooks/pre-commit') failed due to: exit code(1)
  cmdline: /tmp/dvc-test.6040.g8_dftql.64fb4aec-ac74-4af1-9a3e-eec62c6c822a/.git/hooks/pre-commit
  stderr: 'Traceback (most recent call last):
  File "/usr/lib64/python3.7/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib64/python3.7/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib64/python3.7/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/home/ei-grad/repos/github.com/iterative/dvc/dvc/__init__.py", line 61, in <module> 
    __version__ = _update_version_file()
  File "/home/ei-grad/repos/github.com/iterative/dvc/dvc/__init__.py", line 36, in _update_version_file
    dirty = ".mod" if repo.is_dirty() else ""
  File "/home/ei-grad/.virtualenvs/dvc/lib/python3.7/site-packages/git/repo/base.py", line 634, in is_dirty
    len(self.git.diff('--cached', *default_args)):
  File "/home/ei-grad/.virtualenvs/dvc/lib/python3.7/site-packages/git/cmd.py", line 548, in <lambda>
    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)
  File "/home/ei-grad/.virtualenvs/dvc/lib/python3.7/site-packages/git/cmd.py", line 1014, in _call_process
    return self.execute(call, **exec_kwargs)
  File "/home/ei-grad/.virtualenvs/dvc/lib/python3.7/site-packages/git/cmd.py", line 825, in execute 
    raise GitCommandError(command, status, stderr_value, stdout_value)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git diff --cached --abbrev=40 --full-index --raw
  stderr: 'fatal: unable to read 17e0bd067549b04475d7bfe6648ed8fe4f9b0214'
ei-grad commented 5 years ago

Hm... I couldn't reproduce it with the clean pip install -e environment, there is a second condition - to execute the git.commit via GitPython, and it only taking place in tests run and syntetic case like this:

  1. init repo and add test hook 
    git init test1754 && cd test1754
echo -e '#!/bin/sh\npython -c "import dvc"' > .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit

The hook could also be installed by dvc install, the python -c "import dvc" is here just to reduce the scope for reprodution.

  1. make virtualenv and install dvc in editable mode:
virtualenv env
source env/bin/activate
pip install -e git+https://github.com/iterative/dvc.git#egg=dvc
  1. put this into test.py: 
    
from tempfile import mktemp
import os

import git

fname = mktemp(prefix='', dir='') with open(fname, 'w') as f: f.write(fname + '\n')

repo = git.Repo() repo.index.add([fname]) repo.index.commit(fname)


4. and run `python test.py`:

Traceback (most recent call last): File "test.py", line 12, in scm.commit(msg=fname) File "/home/ei-grad/repos/github.com/iterative/dvc/dvc/scm/git.py", line 164, in commit self.git.index.commit(msg) File "/home/ei-grad/test1754/env/lib/python3.7/site-packages/git/index/base.py", line 941, in commit run_commit_hook('pre-commit', self) File "/home/ei-grad/test1754/env/lib/python3.7/site-packages/git/index/fun.py", line 97, in run_commit_hook raise HookExecutionError(hp, cmd.returncode, stderr, stdout) git.exc.HookExecutionError: Hook('/home/ei-grad/test1754/.git/hooks/pre-commit') failed due to: exit code(1) cmdline: /home/ei-grad/test1754/.git/hooks/pre-commit stderr: 'Traceback (most recent call last): File "", line 1, in File "/home/ei-grad/repos/github.com/iterative/dvc/dvc/init.py", line 61, in version = _update_version_file() File "/home/ei-grad/repos/github.com/iterative/dvc/dvc/init.py", line 36, in _update_version_file dirty = ".mod" if repo.is_dirty() else "" File "/home/ei-grad/test1754/env/lib/python3.7/site-packages/git/repo/base.py", line 634, in is_dirty len(self.git.diff('--cached', default_args)): File "/home/ei-grad/test1754/env/lib/python3.7/site-packages/git/cmd.py", line 548, in return lambda args, *kwargs: self._call_process(name, args, kwargs) File "/home/ei-grad/test1754/env/lib/python3.7/site-packages/git/cmd.py", line 1014, in _call_process return self.execute(call, exec_kwargs) File "/home/ei-grad/test1754/env/lib/python3.7/site-packages/git/cmd.py", line 825, in execute raise GitCommandError(command, status, stderr_value, stdout_value) git.exc.GitCommandError: Cmd('git') failed due to: exit code(128) cmdline: git diff --cached --abbrev=40 --full-index --raw stderr: 'fatal: unable to read 4275853377706357a828d4d860a13054c1e9baed' '