Closed webknjaz closed 7 years ago
We use pre-commit install --install-hooks
to accomplish this for our jobs, does this work for you?
Thanks for the suggestion. It doesn't look installing all the dependencies into $HOME/.pre-commit/
env, like pre-commit run
does.
Can you provide some additional output? The two commands run the same code:
You are right, there was no output, because the hooks were installed before that.
Now I've faced another problem: whenever I run pre-commit run --all-files
it tries upgrading them.
I need some way to prevent that. Any ideas?
It shouldn't, can you show some output or something reproducible?
ah I took a peek at your .pre-commit-config.yaml and it's probably because you're using unsupported master
as sha
. You can read more about why this doesn't quite work the way you want it to: https://github.com/pre-commit/pre-commit/issues/158#issuecomment-54103765
The suggested workflow for reproducibility is to use a sha or tag in that field (and periodically upgrade using pre-commit autoupdate
). Actually I think pre-commit autoupdate
will fix your current situation automatically
Oh, I see.. Thanks for explanation.
My goal is to avoid having git installed in container during runtime. Normally I have it as a build dependency only. But it is running git for checks:
An error has occurred: FatalError: git failed. Is it installed, and are you in a Git repository directory?
Check the log at ~/.pre-commit/pre-commit.log
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/pre_commit/git.py", line 20, in get_root
return cmd_output('git', 'rev-parse', '--show-toplevel')[1].strip()
File "/usr/local/lib/python3.5/dist-packages/pre_commit/util.py", line 188, in cmd_output
returncode, cmd, retcode, output=(stdout, stderr),
pre_commit.util.CalledProcessError: Command: ('git', 'rev-parse', '--show-toplevel')
Return code: 1
Expected return code: 0
Output:
Executable `git` not found
Errors: (none)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/pre_commit/error_handler.py", line 47, in error_handler
yield
File "/usr/local/lib/python3.5/dist-packages/pre_commit/main.py", line 166, in main
runner = Runner.create(args.config)
File "/usr/local/lib/python3.5/dist-packages/pre_commit/runner.py", line 28, in create
root = git.get_root()
File "/usr/local/lib/python3.5/dist-packages/pre_commit/git.py", line 23, in get_root
'git failed. Is it installed, and are you in a Git repository '
pre_commit.errors.FatalError: git failed. Is it installed, and are you in a Git repository directory?
Any chance to turn the check off completely? Given that all hooks will already be downloaded and set up at build time.
Ah, pre-commit uses git to both find the top level of the repository and acquire a file list to run hooks against. The failure here is actually while pre-commit is detecting the top level
In other words, git
is an essential runtime dependency even when the hooks are already preinstalled. I think this becomes wontfix?
Well, if it cannot be easily substituted with just a python code, then I think yes. You may close this issue.
Yeah it'd have to reimplement git internals which I'm not really interested in doing :)
@asottile It seems we need to re-open the question. Project workdir in the container is just a folder of the repo at the host machine in RW mode. Thus when it changes the pre-commit hook of the repo it starts pointing to pre-commit inside of the container, which breaks the hook for the host machine. Any ideas? Probably introducing some extra option would help.
This is an inherent problem with prefix installation. If you're using shebangs of executables then the paths need to be the same inside and outside the container. There's not really much that can be done to make them work at different paths since both the environment setup and installation are handled externally (through virtualenv, setuptools, and pip).
If possible, I'd suggest mounting at the same path as you installed the hooks and it should work
Well, I just don't want it to change .git/hooks/pre-commit
while running inside of container. Thus I would prefer some --no-not-change-git-hook
CLI argument or so.
I'll reopen for now, mostly because I'm curious of your workflow. Can you give me a comprehensive list of commands that you're running and whether they're inside / outside of the container as well as the (or an approximation of) the docker command you're running (mostly interested in the mounts). With this I can hopefully suggest either an existing workflow which accomplishes what you want or perhaps a feature if I think it's necessary to do what you want.
For development I use vagga — containerization tool without daemons. Like docker, it also builds and runs LXC containers. But, unlike docker, it runs everything in userspace and isn't meant to be used in production (there's a supervisor lithos, systemd-nspawn and lots of other tools for this).
The config is:
containers:
mysql:
setup:
- !Alpine v3.4
- !Install
- mariadb
- mariadb-client
- !EnsureDir /data
- !EnsureDir /run
environ:
DB_DATABASE: gdg
DB_USERNAME: mysql
DB_PASSWORD: mysql
DB_HOST: 127.0.0.1
DB_PORT: 3307
DB_DATA_DIR: /data
volumes:
/data: !Persistent
name: mysql-data
init-command: _init-mysql
/run: !Tmpfs
mode: 0o766
subdirs:
mysqld:
app:
setup:
- !Ubuntu xenial
- !UbuntuUniverse
- &app-build-deps !BuildDeps
- git
- mercurial
- python3.5-dev
- !Install
- ca-certificates
- python3.5
- !PipConfig
dependencies: true
python-exe: python3.5
- !Depends setup.py
- !Py3Requirements requirements/dev.txt
- !NpmInstall [bower]
- !Sh bower install --allow-root
- !EnsureDir /mnt/db_host
environ-file: /work/.env
test:
setup:
- !Container app
- !BuildDeps *app-build-deps
- !Depends .pre-commit-config.yaml
- !Py3Requirements requirements/test.txt
- !Py3Requirements requirements/test-env.txt
# Git is needed for pre-commit in runtime. Ref:
# github.com/pre-commit/pre-commit/issues/456#issuecomment-269653630
- !Install [git]
# Shadow git-hooks dir, so that pre-commit won't break git hooks in host
# Ref: github.com/pre-commit/pre-commit/issues/456#issuecomment-269856503
- !Sh mount -t tmpfs none /work/.git/hooks
- !Sh HOME=/root pre-commit install --install-hooks
environ:
# Ref:
# github.com/pre-commit/pre-commit-hooks/pull/161#issuecomment-269662841
LANG: en_US.UTF-8
BLUEBERRYPY_CONFIG: "{}"
NOSE_TESTCONFIG_AUTOLOAD_YAML: "config/test/app.yml"
commands:
_init-mysql: !Command
description: Initialize mysql database
container: mysql
run: |
mysql_install_db --datadir=$DB_DATA_DIR
mysqld_safe --user=root --datadir=$DB_DATA_DIR \
--bind-address=$DB_HOST --port=$DB_PORT \
--no-auto-restart --no-watch
while [ ! -S /run/mysqld/mysqld.sock ]; do
sleep .2
done # wait for server to be ready
mysqladmin create $DB_DATABASE
mysql -e "CREATE USER '$DB_USERNAME'@'localhost' IDENTIFIED BY '$DB_PASSWORD';"
mysql -e "GRANT ALL PRIVILEGES ON $DB_DATABASE.* TO '$DB_USERNAME'@'localhost';"
mysql -e "FLUSH PRIVILEGES;"
clean-db: !Command
description: Cleanup mysql database
container: mysql
run: |
mysql_install_db --datadir=$DB_DATA_DIR
mysqld_safe --user=root --datadir=$DB_DATA_DIR \
--bind-address=$DB_HOST --port=$DB_PORT \
--no-auto-restart --no-watch
while [ ! -S /run/mysqld/mysqld.sock ]; do
sleep .2
done # wait for server to be ready
mysqladmin -f drop "$DB_DATABASE"
mysqladmin create "$DB_DATABASE"
mysql -e "GRANT ALL PRIVILEGES ON $DB_DATABASE.* TO '$DB_USERNAME'@'localhost';"
mysql -e "FLUSH PRIVILEGES;"
blueberrypy: !Command
description: |
Run blueberrypy command (you have to provide command and arguments by yourself)
container: app
run: [ blueberrypy ]
mysql: !Command
description: Run RDBMS shell
container: mysql
run: |
mysqld_safe --user=root --datadir=$DB_DATA_DIR \
--bind-address=$DB_HOST --port=$DB_PORT \
--no-auto-restart --no-watch
while [ ! -S /run/mysqld/mysqld.sock ]; do
sleep .2
done
mysql -D $DB_DATABASE
run: !Supervise
description: Run application in development mode
mode: stop-on-failure
children:
run-app: !Command
container: app
run: |
touch /work/.dbcreation # Create lock file
while [ -f /work/.dbcreation ]; do # Acquire lock
sleep .2
done
current_version=$(alembic -c config/alembic.ini -x environment=dev current)
head_version=$(alembic -c config/alembic.ini -x environment=dev heads)
if [ "${current_version}" != "${head_version}" ]; then
alembic -c config/alembic.ini -x environment=dev upgrade head
fi
if [ -z "${current_version}" ]; then
load_gdg_fixtures "$DATABASE_URL" src/GDGUkraine/fixtures/fixtures.yaml || exit 1
fi
blueberrypy serve -b 0.0.0.0:8080
run-db: !Command
container: mysql
run: |
mysqld_safe --user=root --datadir=$DB_DATA_DIR \
--bind-address=$DB_HOST --port=$DB_PORT \
--no-auto-restart --no-watch
while [ ! -S /run/mysqld/mysqld.sock ]; do
sleep .2
done # wait for server to be ready
rm -f /work/.dbcreation # Release lock
while :; do # Emulate infinite loop
sleep 1d;
done
lint: !Command
description: Run linters for gdg.org.ua project
container: test
run: pre-commit run --all-files
'py.test': !Command
description: Run tests for gdg.org.ua project
container: test
run: [py.test, --cov, -v]
test: !Command
description: Run tests for gdg.org.ua project
container: test
run: py.test --cov -v src/tests/
So when I run vagga lint
it:
1) builds app
and test
containers if they don't exist or any of their dependencies in project changed (!Depend
or `requirements.txtetc.)* 1.0) during the build my current directory (with config, which is a repo dir) is being mounted as
/work, rw 1.1) at this stage all needed dependencies are being installed 1.2) during the build I hacked around the issue and shadowed
/work/.git/hookswith
tmpfs, but it doesn't look like a right thing to do 1.3)
HOME=/root pre-commit install --install-hooksinstalls hooks to dir, which is
$HOMEat runtime 2) executes
pre-commit run --all-filescommand inside of
testcontainer 2.0) during the run my project dir is also mounted as
/work`, rw
I'm doing git commit
, which runs pre-commit outside of container, that is why I don't want the build process to influence any git configuration at host.
ok. In that case, I'll add a pre-commit install-hooks
cmdline, I think that's probably the cleanest way to do this
I'll release this in a new version soon. There's some interesting stuff on the horizon I want to get in with this release so it might be a couple days
@asottile great! May I suggest you configure Travis CI for pushing releases to PYPI triggered by pushing new tag? I can send you a PR.
@webknjaz Is the pre-fetching working for you now without .git/ directory inside the container? I'm not sure how pre-commit install-hooks
help to achieve that.
Why wouldn't it? pre-commit manages its checkouts in a dedicated directory under ~/.cache/pre-commit
, this has nothing to do with whether your project is Git-based or not.
I would like to copy only {{.pre-commit-config.yaml}} into the container and then initialize the pre-commit environment.
Here's how I am testing it locally:
$ mkdir foo/
$ cp tat_cdsw/.pre-commit-config.yaml foo/
$ cd foo/
$ pre-commit install
An error has occurred: FatalError: git failed. Is it installed, and are you in a Git repository directory?
please do not bump 4 year old dead threads with off topic questions
you need at least git init .
to run install-hooks
Hi,
I want to suggest introducing a command for downloading all required hooks forcefully. The use case is simple:
I'd like to separate fetching of the hooks and pre-installing them into a build step. This would allow me use caching for containers, saving time for tests.
P.S. Perhaps it's needed to add some argument to
run
command, so that it won't try downloading stuff when executing tests.