search

kubernetes / git-sync

A sidecar app which clones a git repo and keeps it in sync with the upstream.
Apache License 2.0
2.23k stars 411 forks source link

Add git-lfs to image #654

Open jameshearttech opened 1 year ago

jameshearttech commented 1 year ago

I'm requesting git-lfs be added to the image. This issue came up back in 2019 #174. The issue was closed without adding git-lfs to the image. I tested v3.6.2 by adding git-lfs to the image and I'm syncing LFS files from BitBucket without any additional configuration.

This is my dockerfile.

FROM k8s.gcr.io/git-sync/git-sync:v3.6.2
USER root
RUN apt-get update
RUN apt-get -y install git-lfs
USER 65533:65533

This is my container spec within my deployment in k8s.


spec:
  template:
    spec:
      containers:
        - name: git-sync-lfs
          image: example-docker-hub/git-sync-lfs:v3.6.2
          args:
            - -ssh
            - -repo=git@bitbucket.org:example-private-repo/lfs.git
            - -root=/data
            - -dest=lfs
            - -branch=master
            - -depth=1
            - -wait=60
          resources:
            limits:
              cpu: 200m
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: lfs-data
              mountPath: /data
            - name: git-secret
              mountPath: /etc/git-secret
          securityContext:
            runAsUser: 65533
thockin commented 1 year ago

I would want an e2e test for this - how do I force this to be used so I know it works?

jameshearttech commented 1 year ago

Do existing tests not cover this? Can I not just add git-lfs package in a Dockerfile using apt-get then run tests on new image?

My background is sysadmin. I'm new to devops. I know some bash, docker, and k8s. You want me to write a test to prove that it works? Not sure how to do that.

I forked the repo. I'm looking at Dockerfile.in and test_e2e.sh. I'm not sure where to add git-lfs package in Dockerfile.in. Do I pass it an image:tag as an argument to run tests?

thockin commented 1 year ago

If you can show me how to craft an e2e setup which specifically proves LFS works, that would be great. You can see in that file that each testcase is a little snip of shell code which sets up a scenario. What I don't quiote grok is how to set up a repo to use LFS and how to prove that it works.

On Thu, Dec 22, 2022 at 4:56 PM James Ringer @.***> wrote:

Do existing tests not cover this? My background is sysadmin. I'm new to devops. I know some bash. You want me to write a test to prove that it works? Not sure how to do that. I went ahead and forked the repo. I'm looking at test_e2e.sh. How can I help?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

jameshearttech commented 1 year ago

Does this help? GitHub - Configuring Git LFS

GitHub provides some storage and bandwidth for LFS for free. GitHub - About Storage and Bandwidth

We can do a call and work through it together if you'd like. I set it up on BitBucket, but it's probably similar on GitHub. The BitBucket repo I have setup is private, but I could setup a public repo if that helps.

Afaik the only thing that will be different after adding the git-lfs package will be that if a repo contains lfs files they will be synced rather than the text files that point to them. This makes me think existing tests should be fine, but I'm far from and expert.

thockin commented 1 year ago

To do an e2e I need some way to replicate it locally or at least without any auth to an external service. Possible?

On Thu, Dec 22, 2022, 5:23 PM James Ringer @.***> wrote:

Does this help? GitHub - Configuring Git LFS https://docs.github.com/en/repositories/working-with-files/managing-large-files/configuring-git-large-file-storage

We can do a call and work through it together if that helps. I set it up on BitBucket, but it's probably similar on GitHub.

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/git-sync/issues/654#issuecomment-1363494506, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKWAVG5E3GCH32QE7KMKMDWOT5I7ANCNFSM6AAAAAATHFYSJ4 . You are receiving this because you commented.Message ID: @.***>

jameshearttech commented 1 year ago

If you setup a public GitHub repo with LFS I don't think you need authentication.

jameshearttech commented 1 year ago

Do the existing tests use a git repo to test? Maybe just add an LFS file to that repo?

thockin commented 1 year ago

The existing e2e tests use a file:// repo or an SSH repo (still backed by files).

It looks like we can build an LFS test-server binary ( https://github.com/git-lfs/lfs-test-server) to run tests against, but there's not a standard apt pkg for it, so we'll have to pull and build it as part of the test. Look in _test_tools. That's not impossible, it just makes this a little more involved. At least it is written in Go, so we can drop it into tools/go.mod. :)

If you have bandwidth, it's appreciated, otherwise I'll get to this eventually, I hope. :)

On Fri, Dec 23, 2022 at 7:10 AM James Ringer @.***> wrote:

Do the existing tests use a git repo to test? Maybe just add an LFS file to that repo?

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/git-sync/issues/654#issuecomment-1364029692, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKWAVFGCCNKCP5ONWDXENDWOW6E7ANCNFSM6AAAAAATHFYSJ4 . You are receiving this because you commented.Message ID: @.***>

jameshearttech commented 1 year ago

In my fork I have setup lfs-test-server using the sshd folder as an example.

_test_tools
├── lfs-test-server
│   ├── Dockerfile
│   ├── README.md
│   └── run.sh
├── ncsvr
│   ├── Dockerfile
│   ├── README.md
│   └── ncsvr.sh
└── sshd
    ├── Dockerfile
    ├── README.md
    ├── sshd.sh
    └── sshd_config

I am using env vars LFS_USERNAME and LFS_PASSWORD to configure lfs-test-server management username and password. Looking at sshd example I'm assuming we don't want these committed to the repository. Not sure how I'm going to do that. If you don't set these the management interface is not accessible. Also, it looks like you have to login and create a user to use lfs-test-server. Not how I'm going to work around that either. Any ideas?

thockin commented 1 year ago

Ugh, what a cumbersome piece of software to automate. How about something like this Dockerfile:

FROM alpine AS base                                                                                                                                   

RUN mkdir -p /out/etc/apk && cp -r /etc/apk/* /out/etc/apk/
RUN apk add --no-cache --initdb -p /out \
    alpine-baselayout \
    apk-tools \
    curl \
    && true

###############

FROM scratch

ENTRYPOINT []
WORKDIR /

COPY --from=base /out/ /

COPY lfs-test-server /

RUN LFS_ADMINUSER=admin \
    LFS_ADMINPASS=admin \
    timeout 2 \
    /lfs-test-server & \
    sleep 0.5 \
    && curl -s localhost:8080/mgmt/add -u admin:admin -X POST -F name=e2e -F password=e2e

ENTRYPOINT ["/lfs-test-server"]

I did a quick test and that SEEMS to work, as long as lfs-test-server is built with CGO_ENABLED=0 (static binary).

Now you can run the image in docker, get the IP (see ssh examples) and us username "e2e" password "e2e".

jameshearttech commented 1 year ago

I take it from your comment committing the credentials in git is not an issue? I saw in sshd that the certs were in a volume that was not committed.

thockin commented 1 year ago

That's because it's not easy to make "trivial" SSH keys, whereas this case is super easy :)

On Mon, Dec 26, 2022 at 4:34 PM James Ringer @.***> wrote:

I take it from your comment committing the credentials in git is not an issue? I saw in sshd that the certs were in a volume that was not committed.

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/git-sync/issues/654#issuecomment-1365522636, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKWAVFVFQ4LEKCP6F7JCZLWPI2TDANCNFSM6AAAAAATHFYSJ4 . You are receiving this because you commented.Message ID: @.***>

jameshearttech commented 1 year ago

That makes sense. I just wanted to check before moving forward. Btw adding the user with curl that way is just what I was looking for. I'll keep working on this. Thanks.

jameshearttech commented 1 year ago

Haven't circled back to this yet, but I will. Please don't let the issue go stale and close. Thanks!

thockin commented 1 year ago

Hi. I am hoping to have a v4.0.0 release candidate some time in the next month or so - would love to have this in. No pressure, we can always add it later.

jameshearttech commented 1 year ago

I'll put some time toward this over the next few weeks.

jameshearttech commented 1 year ago

I'm working on test_e2e.sh. I added a section to init lfs-test-server.

# Init Git LFS Test Server

# Clone lfs-test-server to _test_tools
git clone https://github.com/git-lfs/lfs-test-server ./_test_tools/lfs-test-server

# Overwrite lfs-test-server Dockerfile
tee ./_test_tools/lfs-test-server/Dockerfile <<EOF
FROM golang:1.18
WORKDIR /go/src/github.com/git-lfs/lfs-test-server
COPY . .
RUN go build
EXPOSE 8080
ENV LFS_ADMINUSER=admin
ENV LFS_ADMINPASS=admin
RUN /go/src/github.com/git-lfs/lfs-test-server/lfs-test-server & \
    sleep 0.5 \
    && curl -s localhost:8080/mgmt/add -u admin:admin -X POST -F name=e2e -F password=e2e
ENTRYPOINT ["/go/src/github.com/git-lfs/lfs-test-server/lfs-test-server"]
EOF

# Build the lfs-test-server image
docker build ./_test_tools/lfs-test-server -t example.com/lfs-test-server:latest

# Run the lfs-test-server container
docker run --rm -d -p 8080:8080/tcp example.com/lfs-test-server:latest

The shell snippet works. Idk if that's what you had in mind. I'm also looking at modifying the init_repo function.

function init_repo() {
    rm -rf "$REPO"
    mkdir -p "$REPO"
    git -C "$REPO" init -q -b "$MAIN_BRANCH"
    touch "$REPO"/file
    git -C "$REPO" add file
    git -C "$REPO" commit -aqm "init file"

    cd "$REPO"
    git lfs install
    git lfs track "*.jpg"
    cd ..
    echo "[lfs]" > "$REPO"/.lfsconfig
    echo "    url = \"http://localhost:8080/\"" >> "$REPO"/.lfsconfig
    git -C "$REPO" add .lfsconfig .gitattributes
    git -C "$REPO" commit -aqm "lfs config"

    touch "$REPO"/lfs.jpg
    git -C "$REPO" add lfs.jpg
    git -C "$REPO" commit -aqm "add lfs.jpg"

    rm -rf "$REPO2"
    cp -r "$REPO" "$REPO2"
}

To confirm the lfs tracked file is sent to lfs-test-server we need to git push, but for that we need a remote.

thockin commented 1 year ago

I don't think you need to do this in the common init_repo, but in the specific LFS test case?

On Sun, Mar 12, 2023, 3:28 AM James Ringer @.***> wrote:

I'm working on test_e2e.sh. I added a section to init lfs-test-server.

Init Git LFS Test Server

Clone lfs-test-server to _test_tools

git clone https://github.com/git-lfs/lfs-test-server ./_test_tools/lfs-test-server

Overwrite lfs-test-server Dockerfile

tee ./_test_tools/lfs-test-server/Dockerfile <<EOF FROM golang:1.18 WORKDIR /go/src/github.com/git-lfs/lfs-test-server COPY . . RUN go build EXPOSE 8080 ENV LFS_ADMINUSER=admin ENV LFS_ADMINPASS=admin RUN /go/src/github.com/git-lfs/lfs-test-server/lfs-test-server & \ sleep 0.5 \ && curl -s localhost:8080/mgmt/add -u admin:admin -X POST -F name=e2e -F password=e2e ENTRYPOINT ["/go/src/github.com/git-lfs/lfs-test-server/lfs-test-server"] EOF

Build the lfs-test-server image

docker build ./_test_tools/lfs-test-server -t example.com/lfs-test-server:latest

Run the lfs-test-server container

docker run --rm -d -p 8080:8080/tcp example.com/lfs-test-server:latest

The shell snippet works. Idk if that's what you had in mind. I'm also looking at modifying the init_repo function.

function init_repo() { rm -rf "$REPO" mkdir -p "$REPO" git -C "$REPO" init -q -b "$MAIN_BRANCH" touch "$REPO"/file git -C "$REPO" add file git -C "$REPO" commit -aqm "init file"

cd "$REPO"
git lfs install
git lfs track "*.jpg"
cd ..
echo "[lfs]" > "$REPO"/.lfsconfig
echo "    url = \"http://localhost:8080/\"" >> "$REPO"/.lfsconfig
git -C "$REPO" add .lfsconfig .gitattributes
git -C "$REPO" commit -aqm "lfs config"

touch "$REPO"/lfs.jpg
git -C "$REPO" add lfs.jpg
git -C "$REPO" commit -aqm "add lfs.jpg"

rm -rf "$REPO2"
cp -r "$REPO" "$REPO2"

}

To confirm the lfs tracked file is sent to lfs-test-server we need to git push, but for that we need a remote.

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/git-sync/issues/654#issuecomment-1465152804, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKWAVAHFJNHWUSATBMSGLDW3WQMDANCNFSM6AAAAAATHFYSJ4 . You are receiving this because you commented.Message ID: @.***>

jameshearttech commented 1 year ago

I've been looking at the repo trying to understand how to do this, but I don't quite follow how everything works. Would it be better if I put the script at /test-lfs.sh?

thockin commented 1 year ago

Do you have a branch pushed to GitHub? I can clone it later this week and see what I see.

On Mon, Mar 13, 2023, 12:00 PM James Ringer @.***> wrote:

I've been looking at the repo trying to understand how to do this, but I don't quite follow how everything works. Would it be better if I put the script at /test-lfs.sh?

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/git-sync/issues/654#issuecomment-1466774042, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKWAVDC4EU4YWEQ22QIPHDW35VF3ANCNFSM6AAAAAATHFYSJ4 . You are receiving this because you commented.Message ID: @.***>

jameshearttech commented 1 year ago

Tbh, there's not much to see at this point. After working on it a bit last weekend I have a better idea of which direction to go. I'm going to work on it again this weekend a bit more, so we'll see how it goes.

jameshearttech commented 1 year ago

I worked with it a bit more and made a little progress. I got a lfs-test-server container along with a git-server-docker container setup and was able to push a commit to git-server-docker containing a jpg that was pushed to lfs-test-server.

I'll throw a bit more time at it next weekend. I feel it's starting to come together, but there is still a ways to go. Ultimately, I want to automate the setup of an environment where we have a repo in git-server-docker and a large file in lfs-test-server such that when we clone the repo with git-sync it pulls the large file with the repo.

Are we on the same page with that goal?

thockin commented 1 year ago

I think that's right - enough to prove that LFS was activated and works.

On Sun, Mar 19, 2023 at 9:40 PM James Ringer @.***> wrote:

I worked with it a bit more and made a little progress. I got a lfs-test-server container along with a git-server-docker container setup and was able to push a commit to git-server-docker containing a jpg that was pushed to lfs-test-server.

I'll throw a bit more time at it next weekend. I feel it's starting to come together, but there is still a ways to go. Ultimately, I want to automate the setup of an environment where we have a repo in git-server-docker and a large file in lfs-test-server such that when we clone the repo with git-sync it pulls the large file with the repo.

Are we on the same page with that goal?

— Reply to this email directly, view it on GitHub https://github.com/kubernetes/git-sync/issues/654#issuecomment-1475611199, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKWAVEEDNZECMWQG2CD7UTW47NTLANCNFSM6AAAAAATHFYSJ4 . You are receiving this because you commented.Message ID: @.***>

thockin commented 1 year ago

Circling around to open topics - any progress here?

jameshearttech commented 1 year ago

I have not had time the past couple of weeks. Tomorrow I'll have some time to pick this up again. Idk how far I'll get, but I'll keep at it until it's done.

jameshearttech commented 1 year ago

I talked with my manager and received approval to start dedicating some of my work time to OSS. This is at the top of my list!

thockin commented 1 year ago

That's awesome! Congrats.

jameshearttech commented 1 year ago

I updated my fork. It's not working yet and it's rough, but if you want to take a look.

jameshearttech commented 1 year ago

@thockin I have not forgotten about this, but I'm on a modified schedule until the end of June with no time for oss. I'll circle back to this in July. Sorry I'm slow and inexperienced.

k8s-triage-robot commented 8 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 6 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 6 months ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/git-sync/issues/654#issuecomment-2014972762): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
jameshearttech commented 6 months ago

Bad bot.

thockin commented 6 months ago

I can re-open if you still want to tackle this, but the bot just sweeps up debris which is often left behind by well-intentioned people who just can't find/make the time to finish (IOW, me, frequently).

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 weeks ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale