Hooks pre-fetching

[webknjaz]

Hi,

I want to suggest introducing a command for downloading all required hooks forcefully. The use case is simple:

1. I pack everything needed for testing into container
2. I deploy that into CI
3. It gets built and tests are ran

I'd like to separate fetching of the hooks and pre-installing them into a build step. This would allow me use caching for containers, saving time for tests.

P.S. Perhaps it's needed to add some argument to run command, so that it won't try downloading stuff when executing tests.

[asottile]

We use pre-commit install --install-hooks to accomplish this for our jobs, does this work for you?

[webknjaz]

Thanks for the suggestion. It doesn't look installing all the dependencies into $HOME/.pre-commit/ env, like pre-commit run does.

[asottile]

Can you provide some additional output? The two commands run the same code:

install --install-hooks

• https://github.com/pre-commit/pre-commit/blob/8837cfa7ffcc419216d4e01392cee0f1ceee9c88/pre_commit/commands/install_uninstall.py#L86

run

• https://github.com/pre-commit/pre-commit/blob/8837cfa7ffcc419216d4e01392cee0f1ceee9c88/pre_commit/commands/run.py#L159
• https://github.com/pre-commit/pre-commit/blob/8837cfa7ffcc419216d4e01392cee0f1ceee9c88/pre_commit/runner.py#L41-L46

[webknjaz]

You are right, there was no output, because the hooks were installed before that.

Now I've faced another problem: whenever I run pre-commit run --all-files it tries upgrading them.

I need some way to prevent that. Any ideas?

[asottile]

It shouldn't, can you show some output or something reproducible?

[asottile]

ah I took a peek at your .pre-commit-config.yaml and it's probably because you're using unsupported master as sha. You can read more about why this doesn't quite work the way you want it to: https://github.com/pre-commit/pre-commit/issues/158#issuecomment-54103765

The suggested workflow for reproducibility is to use a sha or tag in that field (and periodically upgrade using pre-commit autoupdate). Actually I think pre-commit autoupdate will fix your current situation automatically

[webknjaz]

Oh, I see.. Thanks for explanation.

My goal is to avoid having git installed in container during runtime. Normally I have it as a build dependency only. But it is running git for checks:

An error has occurred: FatalError: git failed. Is it installed, and are you in a Git repository directory?
Check the log at ~/.pre-commit/pre-commit.log
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pre_commit/git.py", line 20, in get_root
    return cmd_output('git', 'rev-parse', '--show-toplevel')[1].strip()
  File "/usr/local/lib/python3.5/dist-packages/pre_commit/util.py", line 188, in cmd_output
    returncode, cmd, retcode, output=(stdout, stderr),
pre_commit.util.CalledProcessError: Command: ('git', 'rev-parse', '--show-toplevel')
Return code: 1
Expected return code: 0
Output: 
    Executable `git` not found
Errors: (none)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pre_commit/error_handler.py", line 47, in error_handler
    yield
  File "/usr/local/lib/python3.5/dist-packages/pre_commit/main.py", line 166, in main
    runner = Runner.create(args.config)
  File "/usr/local/lib/python3.5/dist-packages/pre_commit/runner.py", line 28, in create
    root = git.get_root()
  File "/usr/local/lib/python3.5/dist-packages/pre_commit/git.py", line 23, in get_root
    'git failed. Is it installed, and are you in a Git repository '
pre_commit.errors.FatalError: git failed. Is it installed, and are you in a Git repository directory?

Any chance to turn the check off completely? Given that all hooks will already be downloaded and set up at build time.

[asottile]

Ah, pre-commit uses git to both find the top level of the repository and acquire a file list to run hooks against. The failure here is actually while pre-commit is detecting the top level

[asottile]

In other words, git is an essential runtime dependency even when the hooks are already preinstalled. I think this becomes wontfix?

[webknjaz]

Well, if it cannot be easily substituted with just a python code, then I think yes. You may close this issue.

[asottile]

Yeah it'd have to reimplement git internals which I'm not really interested in doing :)

[webknjaz]

@asottile It seems we need to re-open the question. Project workdir in the container is just a folder of the repo at the host machine in RW mode. Thus when it changes the pre-commit hook of the repo it starts pointing to pre-commit inside of the container, which breaks the hook for the host machine. Any ideas? Probably introducing some extra option would help.

[asottile]

This is an inherent problem with prefix installation. If you're using shebangs of executables then the paths need to be the same inside and outside the container. There's not really much that can be done to make them work at different paths since both the environment setup and installation are handled externally (through virtualenv, setuptools, and pip).

If possible, I'd suggest mounting at the same path as you installed the hooks and it should work

[webknjaz]

Well, I just don't want it to change .git/hooks/pre-commit while running inside of container. Thus I would prefer some --no-not-change-git-hook CLI argument or so.

[asottile]

I'll reopen for now, mostly because I'm curious of your workflow. Can you give me a comprehensive list of commands that you're running and whether they're inside / outside of the container as well as the (or an approximation of) the docker command you're running (mostly interested in the mounts). With this I can hopefully suggest either an existing workflow which accomplishes what you want or perhaps a feature if I think it's necessary to do what you want.

[webknjaz]

For development I use vagga — containerization tool without daemons. Like docker, it also builds and runs LXC containers. But, unlike docker, it runs everything in userspace and isn't meant to be used in production (there's a supervisor lithos, systemd-nspawn and lots of other tools for this).

The config is:

containers:
  mysql:
    setup:
      - !Alpine v3.4
      - !Install
        - mariadb
        - mariadb-client
      - !EnsureDir /data
      - !EnsureDir /run
    environ:
      DB_DATABASE: gdg
      DB_USERNAME: mysql
      DB_PASSWORD: mysql
      DB_HOST: 127.0.0.1
      DB_PORT: 3307
      DB_DATA_DIR: /data
    volumes:
      /data: !Persistent
        name: mysql-data
        init-command: _init-mysql
      /run: !Tmpfs
        mode: 0o766
        subdirs:
          mysqld:

  app:
    setup:
      - !Ubuntu xenial
      - !UbuntuUniverse
      - &app-build-deps !BuildDeps
        - git
        - mercurial
        - python3.5-dev
      - !Install
        - ca-certificates
        - python3.5
      - !PipConfig
        dependencies: true
        python-exe: python3.5
      - !Depends setup.py
      - !Py3Requirements requirements/dev.txt
      - !NpmInstall [bower]
      - !Sh bower install --allow-root
      - !EnsureDir /mnt/db_host
    environ-file: /work/.env

  test:
    setup:
      - !Container app
      - !BuildDeps *app-build-deps
      - !Depends .pre-commit-config.yaml
      - !Py3Requirements requirements/test.txt
      - !Py3Requirements requirements/test-env.txt
      # Git is needed for pre-commit in runtime. Ref:
      # github.com/pre-commit/pre-commit/issues/456#issuecomment-269653630
      - !Install [git]
      # Shadow git-hooks dir, so that pre-commit won't break git hooks in host
      # Ref: github.com/pre-commit/pre-commit/issues/456#issuecomment-269856503
      - !Sh mount -t tmpfs none /work/.git/hooks
      - !Sh HOME=/root pre-commit install --install-hooks
    environ:
      # Ref:
      # github.com/pre-commit/pre-commit-hooks/pull/161#issuecomment-269662841
      LANG: en_US.UTF-8
      BLUEBERRYPY_CONFIG: "{}"
      NOSE_TESTCONFIG_AUTOLOAD_YAML: "config/test/app.yml"

commands:
  _init-mysql: !Command
    description: Initialize mysql database
    container: mysql
    run: |
      mysql_install_db --datadir=$DB_DATA_DIR
      mysqld_safe --user=root --datadir=$DB_DATA_DIR \
                  --bind-address=$DB_HOST --port=$DB_PORT \
                  --no-auto-restart --no-watch
      while [ ! -S /run/mysqld/mysqld.sock ]; do
        sleep .2
      done  # wait for server to be ready
      mysqladmin create $DB_DATABASE
      mysql -e "CREATE USER '$DB_USERNAME'@'localhost' IDENTIFIED BY '$DB_PASSWORD';"
      mysql -e "GRANT ALL PRIVILEGES ON $DB_DATABASE.* TO '$DB_USERNAME'@'localhost';"
      mysql -e "FLUSH PRIVILEGES;"
  clean-db: !Command
    description: Cleanup mysql database
    container: mysql
    run: |
      mysql_install_db --datadir=$DB_DATA_DIR
      mysqld_safe --user=root --datadir=$DB_DATA_DIR \
                  --bind-address=$DB_HOST --port=$DB_PORT \
                  --no-auto-restart --no-watch
      while [ ! -S /run/mysqld/mysqld.sock ]; do
        sleep .2
      done  # wait for server to be ready
      mysqladmin -f drop "$DB_DATABASE"
      mysqladmin create "$DB_DATABASE"
      mysql -e "GRANT ALL PRIVILEGES ON $DB_DATABASE.* TO '$DB_USERNAME'@'localhost';"
      mysql -e "FLUSH PRIVILEGES;"
  blueberrypy: !Command
    description: |
      Run blueberrypy command (you have to provide command and arguments by yourself)
    container: app
    run: [ blueberrypy ]

  mysql: !Command
    description: Run RDBMS shell
    container: mysql
    run: |
      mysqld_safe --user=root --datadir=$DB_DATA_DIR \
                  --bind-address=$DB_HOST --port=$DB_PORT \
                  --no-auto-restart --no-watch
      while [ ! -S /run/mysqld/mysqld.sock ]; do
        sleep .2
      done
      mysql -D $DB_DATABASE
  run: !Supervise
    description: Run application in development mode
    mode: stop-on-failure
    children:
      run-app: !Command
        container: app
        run: |
          touch /work/.dbcreation  # Create lock file
          while  [ -f /work/.dbcreation ]; do  # Acquire lock
            sleep .2
          done
          current_version=$(alembic -c config/alembic.ini -x environment=dev current)
          head_version=$(alembic -c config/alembic.ini -x environment=dev heads)
          if [ "${current_version}" != "${head_version}" ]; then
            alembic -c config/alembic.ini -x environment=dev upgrade head
          fi
          if [ -z "${current_version}" ]; then
            load_gdg_fixtures "$DATABASE_URL" src/GDGUkraine/fixtures/fixtures.yaml || exit 1
          fi
          blueberrypy serve -b 0.0.0.0:8080
      run-db: !Command
        container: mysql
        run: |
          mysqld_safe --user=root --datadir=$DB_DATA_DIR \
                      --bind-address=$DB_HOST --port=$DB_PORT \
                      --no-auto-restart --no-watch
          while [ ! -S /run/mysqld/mysqld.sock ]; do
            sleep .2
          done  # wait for server to be ready
          rm -f /work/.dbcreation  # Release lock
          while :; do  # Emulate infinite loop
            sleep 1d;
          done
  lint: !Command
    description: Run linters for gdg.org.ua project
    container: test
    run: pre-commit run --all-files

  'py.test': !Command
    description: Run tests for gdg.org.ua project
    container: test
    run: [py.test, --cov, -v]

  test: !Command
    description: Run tests for gdg.org.ua project
    container: test
    run: py.test --cov -v src/tests/

So when I run vagga lint it: 1) builds app and test containers if they don't exist or any of their dependencies in project changed (!Depend or `requirements.txtetc.)* 1.0) during the build my current directory (with config, which is a repo dir) is being mounted as/work, rw 1.1) at this stage all needed dependencies are being installed 1.2) during the build I hacked around the issue and shadowed/work/.git/hookswithtmpfs, but it doesn't look like a right thing to do 1.3)HOME=/root pre-commit install --install-hooksinstalls hooks to dir, which is$HOMEat runtime 2) executespre-commit run --all-filescommand inside oftestcontainer 2.0) during the run my project dir is also mounted as/work`, rw

I'm doing git commit, which runs pre-commit outside of container, that is why I don't want the build process to influence any git configuration at host.

[asottile]

ok. In that case, I'll add a pre-commit install-hooks cmdline, I think that's probably the cleanest way to do this

[asottile]

I'll release this in a new version soon. There's some interesting stuff on the horizon I want to get in with this release so it might be a couple days

[webknjaz]

@asottile great! May I suggest you configure Travis CI for pushing releases to PYPI triggered by pushing new tag? I can send you a PR.

[chopeen]

@webknjaz Is the pre-fetching working for you now without .git/ directory inside the container? I'm not sure how pre-commit install-hooks help to achieve that.

[webknjaz]

Why wouldn't it? pre-commit manages its checkouts in a dedicated directory under ~/.cache/pre-commit, this has nothing to do with whether your project is Git-based or not.

[chopeen]

I would like to copy only {{.pre-commit-config.yaml}} into the container and then initialize the pre-commit environment.

Here's how I am testing it locally:

$ mkdir foo/
$ cp tat_cdsw/.pre-commit-config.yaml foo/ 
$ cd foo/
$ pre-commit install
An error has occurred: FatalError: git failed. Is it installed, and are you in a Git repository directory?

[asottile]

please do not bump 4 year old dead threads with off topic questions

you need at least git init . to run install-hooks