Support git LFS

[oschwald]

What would you like Renovate to be able to do?

Use renovate with files stored in git lfs.

We use Yarn's offline-mirror feature to vendor the packages. The vendored packages are stored in Git using LFS. When Renovate tries to push the changes, it fails with the message below.

Relevant debug logs

Slightly sanitized error message.

DEBUG: 35 file(s) to commit (repository=group/repo, branch=renovate/js)
DEBUG: Committing files to branch renovate/js (repository=group/repo, branch=renovate/js)
DEBUG: Error commiting files (repository=group/repo, branch=renovate/js)
       "err": {
         "task": {
           "commands": [
             "push",
             "origin",
             "renovate/js:renovate/js",
             "--force",
             "-u",
             "--no-verify",
             "--verbose",
             "--porcelain"
           ],
           "concatStdErr": true,
           "format": "utf-8"
         },
         "message": "Pushing to https://github.company.com/group/repo.git\nPOST git-receive-pack (12865 bytes)\nremote: error: GH008: Your push referenced at least 24 unknown Git LFS objects:        \nremote:     fa64832d250b8a4beea35965a3bb9ab26bddfeb73dbfe836ed61e70e3974fe18        \nremote:     efc11cec699211605376abf2a7aa47b7c7e1854d4428bcef9e67b258918d64d5        \nremote:     8e92658ac0ed2e085bbf11f21bb455d6a9eeae173de21d376e9bbd7105378f1e        \nremote:     ...        \nremote: Try to push them with 'git lfs push --all'.        \nerror: failed to push some refs to 'https://github.company.com/group/repo.git'\n", "stack": "Error: Pushing to https://github.company.com/group/repo.git\nPOST git-receive-pack (12865 bytes)\nremote: error: GH008: Your push referenced at least 24 unknown Git LFS objects:        \nremote:     fa64832d250b8a4beea35965a3bb9ab26bddfeb73dbfe836ed61e70e3974fe18        \nremote:     efc11cec699211605376abf2a7aa47b7c7e1854d4428bcef9e67b258918d64d5        \nremote:     8e92658ac0ed2e085bbf11f21bb455d6a9eeae173de21d376e9bbd7105378f1e        \nremote:     ...        \nremote: Try to push them with 'git lfs push --all'.        \nerror: failed to push some refs to 'https://github.company.com/group/repo.git'\n\n    at GitExecutorChain.onFatalException (/home/user/renovate/node_modules/simple-git/src/lib/runners/git-executor-chain.js:60:87)\n    at GitExecutorChain.<anonymous> (/home/user/renovate/node_modules/simple-git/src/lib/runners/git-executor-chain.js:51:28)\n    at Generator.throw (<anonymous>)\n    at rejected (/home/user/renovate/node_modules/simple-git/src/lib/runners/git-executor-chain.js:6:65)\n    at process._tickCallback (internal/process/next_tick.js:68:7)" }
DEBUG: Passing repository-changed error up (repository=group/repo, branch=renovate/js)
 INFO: Repository has changed during renovation - aborting (repository=group/repo)

[viceice]

For this we would need to make the renovate git layer git-lfs aware.

[viceice]

@oschwald Can you prepare a minimal public reproduction repo? So we have something to test against it.

[oschwald]

Thanks. I made an example repo at https://github.com/oschwald/renovate-lfs-issue.

With this repo, the exact failure is:

DEBUG: Error commiting files (repository=oschwald/renovate-lfs-issue, branch=renovate/js)
       "err": {
         "task": {
           "commands": [
             "push",
             "origin",
             "renovate/js:renovate/js",
             "--force",
             "-u",
             "--no-verify",
             "--verbose",
             "--porcelain"
           ],
           "concatStdErr": true,
           "format": "utf-8"
         },
         "message": "Pushing to https://github.com/oschwald/renovate-lfs-issue.git\nPOST git-receive-pack (4516 bytes)\nremote: error: GH008: Your push referenced at least 11 unknown Git LFS objects:        \nremote:     26acb6450ef03e44b1e1df2457588c0177013c9a6ded0f8e058c5d034a5391e2        \nremote:     47ab5c4571504bdee569f03e3423af5b51aa17d6a94866ddcae353ed2d9033eb        \nremote:     c609324ab889515f2f7354ddcc319b6080c9b76f2ac1441c03da031c85458696        \nremote:     ...        \nremote: Try to push them with 'git lfs push --all'.        \nerror: failed to push some refs to 'https://github.com/oschwald/renovate-lfs-issue.git'\n",
         "stack": "Error: Pushing to https://github.com/oschwald/renovate-lfs-issue.git\nPOST git-receive-pack (4516 bytes)\nremote: error: GH008: Your push referenced at least 11 unknown Git LFS objects:        \nremote:     26acb6450ef03e44b1e1df2457588c0177013c9a6ded0f8e058c5d034a5391e2        \nremote:     47ab5c4571504bdee569f03e3423af5b51aa17d6a94866ddcae353ed2d9033eb        \nremote:     c609324ab889515f2f7354ddcc319b6080c9b76f2ac1441c03da031c85458696        \nremote:     ...        \nremote: Try to push them with 'git lfs push --all'.        \nerror: failed to push some refs to 'https://github.com/oschwald/renovate-lfs-issue.git'\n\n    at GitExecutorChain.onFatalException (/home/greg/MaxMind/renovate-test/node_modules/simple-git/src/lib/runners/git-executor-chain.js:60:87)\n    at GitExecutorChain.<anonymous> (/home/greg/MaxMind/renovate-test/node_modules/simple-git/src/lib/runners/git-executor-chain.js:51:28)\n    at Generator.throw (<anonymous>)\n    at rejected (/home/greg/MaxMind/renovate-test/node_modules/simple-git/src/lib/runners/git-executor-chain.js:6:65)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"
       }

[github-actions[bot]]

Hi there,

The Renovate team needs your help! Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on minimal reproductions to understand what is needed.

We may close the issue if you have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[viceice]

I think this needs a new repo config options to configure files (minimatch / wildcard) to checkout with git lfs. By default git lfs should be disabled.

[github-actions[bot]]

Thank you for providing a reproduction! :tada: :rocket:

The Renovate team will take a look at the reproduction repository. Once we confirm the provided repository reproduces the problem, the label will be changed to reproduction:confirmed.

[sgtsquiggs]

git lfs uses a pre-push git hook, which is disabled by renovate using --no-verify #6521

[amacneil]

Hi friends,

What would it take to get git lfs pull support over the finish line? Can we contribute anything to help?

This is the one feature preventing us from using renovate, and like many others we are on yarn 2 with the yarn binary stored in lfs.

Unfortunately, dependabot doesn’t support yarn 2, and renovate doesn’t support git lfs, so we’re in a bit of a rock and hard place.

[jtbandes]

Hi @rarkins – any chance you can comment on the above? 💕

[rarkins]

@fgblomqvist @viceice could either of you summarize the current status?

[viceice]

We still need to define if it will be always enabled? That way i assume it will always automatically checkout all lfs files. So we don't like to have that.

I think we need to add lfs install to buildpack without full enable it. So install binaries only.

Then we need a configurable list of lfs files, which renovate should checkout from / commit to branch.

[amacneil]

Does anyone need this feature for anything other than yarn 2?

I think it would be sufficient for 99% of users to checkout a default list of files, e.g. git lfs pull --include .yarn yarn.lock **/package-lock.json.

If anyone arrives after that with a request for different files, then it could be made configurable (or just add to the lfs checkout list). I haven't heard any use case for making it configurable yet, but many people (like me) are stuck because of yarn 2.

[viceice]

Instead of fully disabling git lfs by default i like to now some common lfs.fetchexclude and /or lfs.fetchinclude settings for renovate to make this work for most users.

I plan to make those settings user configurable.

[amacneil]

Sounds great - looking forward to it!

[viceice]

So renovate defaults would be:

{
  allowGitLfs: true,
  gitLfsInclude: '.yarn,yarn.lock,**/package-lock.json',
  gitLfsExclude: undefined,
}

allowGitLfs: false would override includes to undefiend and excludes to *.

Any other files to consider by default?

[amacneil]

I think those will cover our use case.

Maybe call the flag enableGitLfs: true or even just gitLfs: true ?

[rarkins]

I'm all for letting self-hosted users get their hands on this first, and we work out how to handle it in the "untrusted" app environment later.

[amacneil]

Is it hard to get working in the shared environment? This doesn't seem like it should be any more or less safe than a regular git checkout.

[viceice]

@amacneil The hosted app has disabled git hooks for security concerns, so yes, it's not as easy to enable lfs there.

[defunctzombie]

Is there still a plan to enable git lfs support for the hosted app? Or is the recommendation to self-host?

[viceice]

Yes, but still needs some time for all preparations.

[gjasny]

FYI: buildpack git-lfs install is broken: https://github.com/containerbase/buildpack/pull/238

[steschi]

What's the current status? What is blocking this? I think yarn-v3 and zero-installs with git-lfs are getting more common but without proper lfs support (at least opt-in or only on self-hosted) these repositories can not take advantage of renovate at all.

[simon-liebehenschel]

It took some working hours for me to setup Renovate with git-lfs support (and a custom Python version), so I would like to share a working configuration with other people.

1. Build a custom Docker image.

   FROM renovate/renovate:34.24
   SHELL ["/bin/bash", "-o", "pipefail", "-c"]
   RUN install-tool python 3.10.8  \
      && install-tool poetry 1.2.2
   USER root
   RUN curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash  \
      && apt-get install -y --no-install-recommends git-lfs=3.2.0 \
      && apt-get -y autoremove && apt-get clean && rm -rf /var/lib/apt/lists/*
   USER 1000

   Note that I install a Python version that I use. If you are fine to use the latest Python (Python 3.11), then change this Dockerfile. Calling USER root is required before calling apt-get or the command will fail due to the lack of permissions. If you do not need custom Python, etc. then just install git-lfs:

   FROM renovate/renovate:34.24
   SHELL ["/bin/bash", "-o", "pipefail", "-c"]
   USER root
   RUN curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash  \
      && apt-get install -y --no-install-recommends git-lfs=3.2.0 \
      && apt-get -y autoremove && apt-get clean && rm -rf /var/lib/apt/lists/*
   USER 1000

2. Set required settings to support git-lfs.

   # This is mandatory for git-lfs or it will fail with the `GitLab: LFS objects are missing. Ensure LFS is properly set up or try a manual "git lfs push --all"`
   RENOVATE_GIT_NO_VERIFY: "[]"
   # Perhaps `RENOVATE_BINARY_SOURCE` is redundant if you already installed all required packages in your custom Docker image.
   RENOVATE_BINARY_SOURCE: "install"

   Other configuration options are omitted for the sake of brevity.

This Docker image and config were tested at GitLab SaaS.

[jtbandes]

FYI, for those using yarn, dependabot now supports yarn berry and git lfs (https://github.com/dependabot/dependabot-core/pull/5833).

[rarkins]

@AIGeneratedUsername thanks for posting your solution. I have some questions on some of the things you added:

RUN install-tool python 3.10.8  \
    && install-tool poetry 1.2.2

I'm wondering why these are necessary. Shouldn't both of these be dynamically installed due to binarySource=install at runtime?

RENOVATE_BINARY_SOURCE: "install"

First of all, I'm not sure if this is mandatory for Git LFS in any way. But also, it is now the default behavior for renovate/renovate non-slim images anyway.

The main one of interest to me is:

RENOVATE_GIT_NO_VERIFY: "[]"

Here is the option for this:

  {
    name: 'gitNoVerify',
    description:
      'Which Git commands will be run with the `--no-verify` option.',
    type: 'array',
    subType: 'string',
    allowString: true,
    allowedValues: ['commit', 'push'],
    default: ['commit', 'push'],
    stage: 'global',
    globalOnly: true,
  },

So by setting this you're making sure Renovate runs e.g. git push --no-verify. Is this may be a workaround and not yet a solution?

[viceice]

disable git no-verify is required because LFS is working via git hooks

[rarkins]

Would we have any way in future to detect LFS is in use and remove --no-verify automatically?

[viceice]

not easily, we also set it to suppress arbitrary git hooks from repos for security reasons.

[mat813]

I started using SheetJS in a projet, and they recommend vendoring the package file because the npm registry has an outdated one, as it is a bit large, I used git lfs to store it, and now renovate cannot run yarn install, it errors out with:

error Extracting tar content of undefined failed, the file appears to be corrupt: "Unexpected end of data"

Also, we're considering moving from yarn classic to yarn v3 using zero installs which kinda requires git-lfs if you don't want your repository size to explode. But we are blocked by renovate not supporting lfs.

[quine]

+1 for supporting Git LFS. We're using self-hosted Renovate, and we've got several Go repos with vendored modules (with files exceeding our GHE file size limit), thus requiring Git LFS.

[small-stone]

I have same problem:

WARN: Error updating branch (repository=ee-frontend, branch=branch-testfive/major-react-router-monorepo) "err": { "task": { "commands": ["checkout", "-f", "master", "--"], "format": "utf-8", "parser": "[function]" }, "message": "Your branch is up to date with 'origin/master'.\nAlready on 'master'\n\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting '.git/hooks/post-checkout'.\n\n", "stack": "Error: Your branch is up to date with 'origin/master'.\nAlready on 'master'\n\nThis repository is configured for Git LFS but 'git-lfs' was not found on your path. If you no longer wish to use Git LFS, remove this hook by deleting '.git/hooks/post-checkout'.\n\n\n at Object.action (/opt/containerbase/tools/renovate/36.57.2/node_modules/simple-git/src/lib/plugins/error-detection.plugin.ts:42:29)\n at PluginStore.exec (/opt/containerbase/tools/renovate/36.57.2/node_modules/simple-git/src/lib/plugins/plugin-store.ts:29:29)\n at /opt/containerbase/tools/renovate/36.57.2/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:127:42\n at new Promise ()\n at GitExecutorChain.handleTaskData (/opt/containerbase/tools/renovate/36.57.2/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:124:14)\n at GitExecutorChain. (/opt/containerbase/tools/renovate/36.57.2/node_modules/simple-git/src/lib/runners/git-executor-chain.ts:100:40)\n at Generator.next ()\n at fulfilled (/opt/containerbase/tools/renovate/36.57.2/node_modules/simple-git/dist/cjs/index.js:55:24)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)" }

I fork renovate-runner on my gitlab, and use gitlab runner to run renovate with docker like ghcr.io/renovatebot/renovate:36.57.2@xxx

So I urgently need support Git LFS

[mat813]

Maybe it would be possible, for example, to start by adding git-lfs to the full image ? That way it would be available, and people don't have to maintain custom renovate images.

[viceice]

simply install git-lfs on the fly before running renovate on your ci and configure gitNoVerify

https://github.com/containerbase/base/blob/6885c851db8d7f40aaea33b9c2fabc4652958e89/test/latest/Dockerfile#L162

https://docs.renovatebot.com/self-hosted-configuration/#gitnoverify

[ekristen]

I hit a problem with no git-lfs being install in the container with golang this week.

TL;DR - I modified the docker run locally to test, install-tool followed by git lfs install --skip-repo (to install it for the user) made everything work as expected again.

https://github.com/renovatebot/renovate/discussions/25041#discussioncomment-7203684

[rarkins]

@viceice if we merely install git-lfs in the images, will that cause any problems?

[ekristen]

I think you'd have to install the tool and also initialize it.

[ekristen]

install-tool git-lfs v3.4.0

[viceice]

@viceice if we merely install git-lfs in the images, will that cause any problems?

no

@ekristen the install tool command will already initialize https://github.com/containerbase/base/blob/80f6e73580b58d6ed7e9c708424124f3fbfb23b2/src/usr/local/containerbase/tools/v2/git-lfs.sh#L40-L44

[rarkins]

@viceice we'd need it installed in the slim, full and sidecar images, right?

[viceice]

not on sidecar.

we didn't yet, because there is no way for renovate to block it in the app

[cyrilc-pro]

FYI, one issue with Git LFS is that Renovate is running commands like git show origin/master:package-lock.json. This command is OK as long as package-lock.json is not stored using Git LFS.

[ekristen]

@cyrilc-pro that's a fair point, but if it is store in git-lfs today, things will not work properly so it just might have to be a known limitation in the documentation.

[simon-liebehenschel]

FYI, one issue with Git LFS is that Renovate is running commands like git show origin/master:package-lock.json. This command is OK as long as package-lock.json is not stored using Git LFS.

@cyrilc-pro Could you please clarify what exactly will be "not OK" if a file is stored using Git LFS? Yes, the mentioned command will not display a file content, but what exactly will go wrong in Renovate?

[cyrilc-pro]

@ekristen I'm OK with this being a limitation. In my case, I have a package-lock.json that is almost 1.5MB, which is why our company policy requires it to be stored with Git LFS.

@AIGeneratedUsername The symptom is that the package-lock.json is not updated along with the package.json. The Renovate logs contain a warning about the package-lock.json not being valid JSON:

WARN: Error parsing npm lock file (repository=<redacted>, branch=<redacted>-npm-minor)
      "err": {
        "message": "Unexpected token v in JSON at position 0",
        "stack": "SyntaxError: Unexpected token v in JSON at position 0\n    at JSON.parse (<anonymous>)\n    at parseLockFile (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/modules/manager/npm/utils.ts:10:27)\n    at writeExistingFiles (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/modules/manager/npm/post-update/index.ts:161:24)\n    at getAdditionalFiles (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/modules/manager/npm/post-update/index.ts:473:3)\n    at processBranch (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/workers/repository/update/branch/index.ts:471:31)\n    at writeUpdates (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/workers/repository/process/write.ts:159:17)\n    at update (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/workers/repository/process/extract-update.ts:214:11)\n    at Object.renovateRepository (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/workers/repository/index.ts:76:19)\n    at attributes.repository (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/workers/global/index.ts:184:11)\n    at start (/opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/workers/global/index.ts:169:7)\n    at /opt/containerbase/tools/renovate/37.56.3/node_modules/renovate/lib/renovate.ts:18:22"
      }

The ultimate result is the failure of the npm install command on the branch that Renovate created.

[natan500]

Hi, I have the same issue as @cyrilc-pro mentioned, is there any solution for that?

[ekristen]

At this point if package-lock.json or any file is in git-lfs the git show will not work, but as it stands right now, because they are in git-lfs, and since renovate doesn't support git-lfs, renovate shouldn't be working at all anyways.

I propose we get git-lfs support merged and then we can look to improve the git show command, theoretically you can do git show origin/master:npm-package.json | git lfs smudge

So the code could be changed to do git show and look if version is present with git-lfs header and if so run it through git lfs smudge to get the actual file contents.

[ekristen]

Any thoughts on pushing this forward, I host documentation with my source code as such I have some jpegs and the like stored in git-lfs, this is breaking golang dependencies because the hashes don't match. In my experience supporting git-lfs is fairly trivial. Thanks all!

[rarkins]

I've lost track of what the next steps are

[viceice]

we don't support it because we blocked git hooks for security reasons.

if you self-host, then you can configure to allow hooks and you need to install git-lfs

here's how to do in official renovate images: https://github.com/containerbase/base/blob/821a5931113d622de322683957f0c435352f703c/test/latest/Dockerfile#L170