Support git LFS

[oschwald]

What would you like Renovate to be able to do?

Use renovate with files stored in git lfs.

We use Yarn's offline-mirror feature to vendor the packages. The vendored packages are stored in Git using LFS. When Renovate tries to push the changes, it fails with the message below.

Relevant debug logs

Slightly sanitized error message.

DEBUG: 35 file(s) to commit (repository=group/repo, branch=renovate/js)
DEBUG: Committing files to branch renovate/js (repository=group/repo, branch=renovate/js)
DEBUG: Error commiting files (repository=group/repo, branch=renovate/js)
       "err": {
         "task": {
           "commands": [
             "push",
             "origin",
             "renovate/js:renovate/js",
             "--force",
             "-u",
             "--no-verify",
             "--verbose",
             "--porcelain"
           ],
           "concatStdErr": true,
           "format": "utf-8"
         },
         "message": "Pushing to https://github.company.com/group/repo.git\nPOST git-receive-pack (12865 bytes)\nremote: error: GH008: Your push referenced at least 24 unknown Git LFS objects:        \nremote:     fa64832d250b8a4beea35965a3bb9ab26bddfeb73dbfe836ed61e70e3974fe18        \nremote:     efc11cec699211605376abf2a7aa47b7c7e1854d4428bcef9e67b258918d64d5        \nremote:     8e92658ac0ed2e085bbf11f21bb455d6a9eeae173de21d376e9bbd7105378f1e        \nremote:     ...        \nremote: Try to push them with 'git lfs push --all'.        \nerror: failed to push some refs to 'https://github.company.com/group/repo.git'\n", "stack": "Error: Pushing to https://github.company.com/group/repo.git\nPOST git-receive-pack (12865 bytes)\nremote: error: GH008: Your push referenced at least 24 unknown Git LFS objects:        \nremote:     fa64832d250b8a4beea35965a3bb9ab26bddfeb73dbfe836ed61e70e3974fe18        \nremote:     efc11cec699211605376abf2a7aa47b7c7e1854d4428bcef9e67b258918d64d5        \nremote:     8e92658ac0ed2e085bbf11f21bb455d6a9eeae173de21d376e9bbd7105378f1e        \nremote:     ...        \nremote: Try to push them with 'git lfs push --all'.        \nerror: failed to push some refs to 'https://github.company.com/group/repo.git'\n\n    at GitExecutorChain.onFatalException (/home/user/renovate/node_modules/simple-git/src/lib/runners/git-executor-chain.js:60:87)\n    at GitExecutorChain.<anonymous> (/home/user/renovate/node_modules/simple-git/src/lib/runners/git-executor-chain.js:51:28)\n    at Generator.throw (<anonymous>)\n    at rejected (/home/user/renovate/node_modules/simple-git/src/lib/runners/git-executor-chain.js:6:65)\n    at process._tickCallback (internal/process/next_tick.js:68:7)" }
DEBUG: Passing repository-changed error up (repository=group/repo, branch=renovate/js)
 INFO: Repository has changed during renovation - aborting (repository=group/repo)

[ekristen]

@viceice so is this something that's not going to be supported in the hosted version?

[viceice]

• https://github.com/renovatebot/renovate/pull/10748

this was a starter to fully support lfs by renovate. feel free to take and complete it 🤗

[rarkins]

Not in the foreseeable future for security reasons

[viceice]

@viceice so is this something that's not going to be supported in the hosted version?

probably yes, until someone finds a way without git hooks.

[ekristen]

Thanks @rarkins and @viceice -- I'm pretty familiar with git-lfs but I went and did a bit of reading. You said not going to be allowed due to hooks being not supported for security reasons, but as far as I know git-lfs doesn't actually use any hooks, at least not the traditional .git/hooks.

If we were to install git-lfs for the user running renovate and more specifically the git commands, the git lfs install simply adds a filter stanza to the users ~/.gitconfig which is then used during clone and checkout to read the .gitattributes and look for matching filters to do the git-lfs things which to my knowledge filters is in internal git supported mechanism.

I've checked my configuration, and I'm looking at some additional things right now but I can't find any use of hooks or reference to hooks. Am I missing something?

[viceice]

it's using hooks somewhere to push the lfs files, see comments above. that's why you can allow hooks for self-host

[ekristen]

@viceice ah, that's the missing link, for pushing, I see a post-checkout too (now that I'm looking a bit deeper), wonder if they are used for initial cloning though.

What if we supported git-lfs for cloning but not for pushing, make that a limitation. Then for cloning it would work for go mods and others that have hashes of their contents but wouldn't support renovating any files that are actually in git-lfs?

Another approach could be to take the direnv.net approach and allow only hooks that meet a specific hash. For example, we could allow the 3 hook files that get created so long as their hash matches.

#!/bin/sh
command -v git-lfs >/dev/null 2>&1 || { echo >&2 "\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting '.git/hooks/pre-push'.\n"; exit 2; }
git lfs pre-push "$@"

This is the pre-push in a cloned repo that has git-lfs file, renovate could allow this file to exist in the repo as long as the file matches 911a6ef2ce82426869f449da86b8ca5f7cb6205dfd8236b35ef277b26aef9fed if it doesn't match it's removed?

Thoughts?

[viceice]

@rarkins i think we need to revert git-lfs install. otherwise the hosted app will start downloading all lfs files on checkout.

we need a very defensive setting first: https://github.com/renovatebot/renovate/pull/10748