search

milkey-mouse / git-lfs-s3-proxy

Store Git LFS files on any service with an S3 API
https://git-lfs-s3-proxy.pages.dev
Creative Commons Zero v1.0 Universal
45 stars 37 forks source link

Does this support setting up git LFS on gitlab repositories with exsisting lfs storage? #8

Closed Nbc66 closed 8 months ago

Nbc66 commented 8 months ago

As the title says

Does this support gitlab aswell so far it says im missing objects from lfs when i try to use this on a gitlab repo

Nbc66 commented 8 months ago

ok i actualy figgured it out you need to dissable gitlab lfs storage on your project using the gitlab api

example CURL command you can send

curl --request PUT --header "PRIVATE-TOKEN: <your-token>" \
     --url "https://gitlab.com/api/v4/projects/<your-project-ID>" \
     --data "lfs_enabled=false" # to turn off: "lfs_enabled=false"

NOTE: You need to set this after you have setup your own lfs storage on r2 or any other service

milkey-mouse commented 8 months ago

Thanks for investigating this. Would you be willing to try the following for me? I want to see if the "Upload existing LFS objects" instructions might work with a minor modification.

Nbc66 commented 8 months ago

Allredy tested it that way you need to dissable it beffore you can push all the lfs objects otherwise gitlab will give you ore commit hook fails telling you you are missing objects

Nbc66 commented 8 months ago

I also made a pull request that addresses this issue and updated the readme

9

Nbc66 commented 8 months ago

Thanks for investigating this. Would you be willing to try the following for me? I want to see if the "Upload existing LFS objects" instructions might work with a minor modification.

  • Create a GitLab repo with their default LFS server enabled
  • Track some file with Git LFS
  • Commit and push the file
  • Remove the file in a subsequent commit (master/main should no longer include the file).

  • On a new clone of the repo:

    • Run git lfs fetch --all.
    • Set up git-lfs-s3-proxy as described in the README. Don't disable GitLab LFS though.
    • Run git lfs push --all.
  • Clone the repo one more time. Let's see if on this new clone you get the "missing objects from LFS" message.

Ok im dumb i just realised what you are talking about as in deleteing the lfs objects

Nbc66 commented 8 months ago

Ok i gota mentione that our lfs repo is over 19gb so it takes time for me to clone it fully it will be a while beffore i can test this

Nbc66 commented 8 months ago

Also you must clone the repo using git clone --bare Then do git lfs fetch --all this needs to be done beffore you actually dissable the gitlab lfs so we can pull all the objcets from gitlabs server then dissable the lfs option on gitlab setup the lfs proxy and then you can push to it. Using with git lfs fetch --all will give you all the lfs objects when fetching with --all

Nbc66 commented 8 months ago

Thanks for investigating this. Would you be willing to try the following for me? I want to see if the "Upload existing LFS objects" instructions might work with a minor modification.

  • Create a GitLab repo with their default LFS server enabled
  • Track some file with Git LFS
  • Commit and push the file
  • Remove the file in a subsequent commit (master/main should no longer include the file).

  • On a new clone of the repo:

    • Run git lfs fetch --all.
    • Set up git-lfs-s3-proxy as described in the README. Don't disable GitLab LFS though.
    • Run git lfs push --all.
  • Clone the repo one more time. Let's see if on this new clone you get the "missing objects from LFS" message.

tested it I still get the lfs objects missing issue

milkey-mouse commented 8 months ago

Fixed by #9. Thanks!

Nbc66 commented 8 months ago

so about this, the use of git clone --bare is if you have more than 1 branch on a repository if you do git lfs fetch --all on a normal clone it will only fetch objects relevant to that specific branch for some reason

so using git clone --bare it will allow you to fetch the lfs objects for all branches and tags and it should fetch every object

this should be only said to be done if the repository with already existing lfs object has multiple branches or tags

Nbc66 commented 8 months ago

you can read more about it here

https://github.com/git-lfs/git-lfs/discussions/4984#discussioncomment-2676826 @milkey-mouse