Closed eugeneYWang closed 1 year ago
BTW, if I understand the README of this repo correctly, the way I mounted two volumns as two sub-folders under /git
is fine, right?
regards.
Update:
I have made a workaround to populate an empty known_hosts
file. Here is my custom docker file.
FROM k8s.gcr.io/git-sync/git-sync:v3.6.4
USER root
RUN mkdir -p /etc/git-secret/
RUN ssh-keyscan github.com > /etc/git-secret/known_hosts
USER 65533:65533
And this is the latest git-sync service:
git-sync-service:
# image: k8s.gcr.io/git-sync/git-sync:v3.6.4
build:
context: ./vendor-dockerfile/git-sync
dockerfile: Dockerfile
user: "${AIRFLOW_UID:-50000}:0"
profiles:
- sync-dag
environment:
- GIT_SYNC_REPO=${GIT_SYNC_DAG_REPO}
- GIT_SYNC_BRANCH=${GIT_SYNC_DAG_BRANCH}
# -1 below means it will retry forever. TODO: discuss this value for UAT setup
# Note: decide that maybe one-time sync is better for local setup
# - GIT_SYNC_MAX_FAILURES = -1
- GIT_SYNC_ROOT=/git
- GIT_SYNC_SSH=true
# - GIT_SYNC_KNOWN_HOSTS=false
- GIT_SYNC_SSH_KEY_FILE=/etc/git-secret/ssh
- GIT_SYNC_ONE_TIME=true
- GIT_SYNC_DEPTH=1
volumes:
# - git-sync-root-volumn:/git
- ./dags:/git/dags
- ./plugins:/git/plugins
- ${GIT_SSH_KEY_FILE_PATH}:/etc/git-secret/ssh
# other default values: 10s for sync interval($GIT_SYNC_PERIOD), HEAD for checkout revision($GIT_SYNC_REV)
volumes:
postgres-db-volume:
mysql-db-volumn:
# git-sync-root-volumn:
With the updated context, my new problem is:
How may I map my external dirs ./dags
, ./plugins
into two sub-dirs in the repo /git/dags
, /git/plugins
?
The problem returns:
docker-compose-git-sync-service-1 | INFO: detected pid 1, running init handler
docker-compose-git-sync-service-1 | I0301 00:48:41.924057 12 main.go:389] "level"=0 "msg"="starting up" "pid"=12 "args"=["/git-sync"]
docker-compose-git-sync-service-1 | I0301 00:48:41.938694 12 main.go:934] "level"=0 "msg"="cloning repo" "origin"="git@github.com:(my team)/(dag-repo).git" "path"="/git"
docker-compose-git-sync-service-1 | I0301 00:48:41.941094 12 main.go:940] "level"=0 "msg"="git root exists and is not empty (previous crash?), cleaning up" "path"="/git"
docker-compose-git-sync-service-1 | E0301 00:48:41.944425 12 main.go:535] "msg"="too many failures, aborting" "error"="unlinkat /git/dags: permission denied" "failCount"=1
docker-compose-git-sync-service-1 exited with code 1
I found a way to pull git-repo content down . However, the symlink created by git-sync
has made Airflow scheduler fallen into a recursive loop of seeing the rev folder and the symlink re-directing back to the rev folder.
In this way, the output of git-sync cannot be used by Airflow directly in the context of docker compose.
Do you have any suggestion for my use case?
Attaching a working docker-compose service:
# have not tested for scenes when ./dags ./plugins are not created.
# prerequisite:
# a ssh keypair file on the host path
git-sync-service:
# image: k8s.gcr.io/git-sync/git-sync:v3.6.4
build:
context: ./vendor-dockerfile/git-sync
dockerfile: Dockerfile
user: "${AIRFLOW_UID:-50000}:0"
profiles:
- sync-dag
environment:
- GIT_SYNC_REPO=${GIT_SYNC_DAG_REPO}
- GIT_SYNC_BRANCH=${GIT_SYNC_DAG_BRANCH}
- GIT_SYNC_ADD_USER=true
# -1 below means it will retry forever. TODO: discuss this value for UAT setup
# Note: decide that maybe one-time sync is better for local setup
# - GIT_SYNC_MAX_FAILURES = -1
- GIT_SYNC_ROOT=/tmp
- GIT_SYNC_SSH=true
- GIT_SYNC_SSH_KEY_FILE=/etc/git-secret/ssh
- GIT_SYNC_ONE_TIME=true
- GIT_SYNC_DEPTH=0
volumes:
# - git-sync-root-volumn:/dags
- ./dags:/tmp
# - ./plugins:/git/plugins
- ${GIT_SSH_KEY_FILE_PATH}:/etc/git-secret/ssh
and https://github.com/kubernetes/git-sync/issues/314#issuecomment-740781574
above two links would help me do the post jobs!
Any more tips about that hook script is appreciated!
I set - GIT_SYNC_KNOWN_HOSTS=false
In v3 this is called GIT_KNOWN_HOSTS
- it's a "bug" of sorts that the name is inconsistent. Flags are better than env vars, IMO.
WRT the rest, there are a lot of questions being asked at the same time.
git-sync needs to own whatever directory you give it as the root. The root itself might be a volume, but git-sync wants to own everything in it, which means you can't put other volumes underneath it.
git-sync always "publishes" a synced repo via a symlink. You can put thsat symlink anywhere that git-sync is able to write - for example, you can use "/tmp/git-sync" as the --root
, and "/git/dags" as the --dest
. /git/dags will be symlink to /tmp/git-syc/<something you should not hardcode>
which is the "current" state of the git repo. If you readlink
the link and take the basename
, it will give you the git SHA.
I'm going to close this - I don't think it's a "bug" - but I am happy to continue to discuss how to solve the usage pattern you want
@thockin Thank you for giving tips.
As of my usage pattern, I think I will have to hack my own way using the exechook_command option, to copy the git data to my mounted host column.
just as https://github.com/kubernetes/git-sync/issues/314, we need actual file and a symlink is not going to work in the context of docker compose. I tried to use a named docker volumn as the root dir of git-sync, but I got blocked like https://github.com/kubernetes/git-sync/issues/245, even if I found a stack overflow post that is supposed to solve it. But it still does not work for me.
The git-sync container is designed to do the right thing by default wrt volumes - you should only have issues if you change the user it runs as or the volume mount path, in which case there's not much it can do automatically.
@thockin pls consider reopening. as 3.6.5
ignores the flag for the setting instead of env:
INFO: detected pid 1, running init handler
I0413 14:28:01.318302 13 main.go:401] "level"=0 "msg"="starting up" "pid"=13 "args"=["/git-sync","--repo=git@github.com:myrepo","--ssh","true","--branch=master","--ssh-known-hosts=false"]
ERROR: can't configure SSH: can't access SSH known_hosts: stat /etc/git-secret/known_hosts: no such file or directory
EDIT - tested different behavior:
git-sync
effectively ignores the known hosts when passing the corresponding env
docker run -d \
-v $DIR:/tmp/git \
-v ssh:/etc/git-secret/ssh \
-u$(id -u):$(id -g) \
-e GIT_KNOWN_HOSTS=false \
registry.k8s.io/git-sync/git-sync:v3.6.5 \
--repo=git@github.com:myrepo \
--ssh true
but getting reported error when using the flag
docker run -d \
-v $DIR:/tmp/git \
-v ssh:/etc/git-secret/ssh \
-u$(id -u):$(id -g) \
registry.k8s.io/git-sync/git-sync:v3.6.5 \
--repo=git@github.com:myrepo \
--ssh true \
--ssh-known-hosts false
I can't reproduce a failure:
X=/tmp/$RANDOM
rm -rf $X
mkdir -p $X
docker run -ti \
-v $X:/tmp/git \
-v ~/.ssh/id_ed25519:/etc/git-secret/ssh:ro \
-u$(id -u):$(id -g) \
registry.k8s.io/git-sync/git-sync:v3.6.5 \
--repo=git@github.com:kubernetes/git-sync \
--add-user \
--ssh \
--ssh-known-hosts=false \
-v 2
INFO: detected pid 1, running init handler
I0413 19:02:36.314066 12 main.go:401] "level"=0 "msg"="starting up" "pid"=12 "args"=["/git-sync","--repo=git@github.com:kubernetes/git-sync","--add-user","--ssh","--ssh-known-hosts=false","-v","2"]
I0413 19:02:36.336143 12 main.go:1183] "level"=1 "msg"="setting up git SSH credentials"
I0413 19:02:36.336179 12 main.go:539] "level"=1 "msg"="syncing repo"
I0413 19:02:36.336200 12 main.go:950] "level"=0 "msg"="cloning repo" "origin"="git@github.com:kubernetes/git-sync" "path"="/tmp/git"
I0413 19:02:38.765109 12 main.go:760] "level"=0 "msg"="syncing git" "rev"="HEAD" "hash"="bb0128b883a7d8a48d04f9de714803afd24510e9"
I0413 19:02:38.770119 12 main.go:749] "level"=1 "msg"="removing worktree" "path"="/tmp/git/bb0128b883a7d8a48d04f9de714803afd24510e9"
I0413 19:02:38.776632 12 main.go:800] "level"=0 "msg"="adding worktree" "path"="/tmp/git/bb0128b883a7d8a48d04f9de714803afd24510e9" "branch"="origin/master"
I0413 19:02:38.853862 12 main.go:860] "level"=0 "msg"="reset worktree to hash" "path"="/tmp/git/bb0128b883a7d8a48d04f9de714803afd24510e9" "hash"="bb0128b883a7d8a48d04f9de714803afd24510e9"
I0413 19:02:38.853879 12 main.go:865] "level"=0 "msg"="updating submodules"
I0413 19:02:38.884503 12 main.go:717] "level"=1 "msg"="creating tmp symlink" "root"="/tmp/git/" "dst"="bb0128b883a7d8a48d04f9de714803afd24510e9" "src"="tmp-link"
I0413 19:02:38.885478 12 main.go:722] "level"=1 "msg"="renaming symlink" "root"="/tmp/git/" "old_name"="tmp-link" "new_name"="git-sync"
I0413 19:02:38.891484 12 main.go:608] "level"=1 "msg"="next sync" "wait_time"=1000000000
Hmm, changing it to --ssh-known-hosts false
(space rather than =) fails. Debugging, that looks like a flag parsing failure.
Use =
when setting values on flags, I guess? git-sync v3 uses Go's standard flag parsing library. It looks like even pflag has this oddity - boolean flags don't NORMALLY take an argument, so they only work with =/
Thanks @thockin ! I use env
eventually so no showstopper but for debugging permission issues (logs not explicit, even verbose), I docker run with entrypoint override to sh
, then I like to use git-sync
command with flags. Just for my info, is v4 different ?
Unfortunately not. A bool flag is usually used as --boolflag
, meaning ["--boolflag=true"]
, so the flag parser can't REALLY tell whether --boolflag false
means ["--boolflag=false"]
or ["--boolflag=true", "false"]
. Both go flags and pflags chose the latter.
In general, the =
syntax is always safe.
Problem:
In a compose file that does not worry about middle-man-attack, I set
- GIT_SYNC_KNOWN_HOSTS=false
, but the docker still complains messages likeContext:
I am using this docker image as a service in a docker compose file along with other airflow services. In order to sync Dag folder and other folders with a dedicated repo.
This is the git-sync service in my compose file: