Disk Space Error Preventing Onboarding PR (Github, 0% cloned)

[cpsauer]

Foremost, thanks all for making such a great thing for the world :) Unfortunately, I ran into an issue trying to get things off the ground in my repo. Details below:

What Renovate type, platform and version are you using?

GitHub App. Logs show v23.88.3.

Describe the bug

I added Renovate to a private repo, but all Renovate jobs have errored out, so the onboarding PR never arrived. In all cases, logs show a disk space error, but also that it's occurring while the repo is still at 0% cloned (?).

(The repo is fairly large, but not enormous. ~9GB shallow. It seems to me like repo size probably isn't the issue, though, because the logs are showing 0% cloning completion at the time of failure. But it's possible.)

Relevant debug logs

All hourly jobs seem to be failing the same way from the logs. If helpful to Whitesource folks: this is from the first job, ID: 266769610.

The logs start the same way: Renovate version, no cache, endpoint, bunch of checks, etc. Then:

DEBUG: Initializing git repository into /mnt/renovate/gh/hedronvision/Hedron
DEBUG: git clone error
{
  "err": {
    "task": {
      "commands": [
        "clone",
        "--depth=10",
        "https://**redacted**@github.com/<ORG>/<REPO>.git",
        "."
      ],
      "format": "utf-8"
    },
    "message": "Cloning into '.'...\nUpdating files:   0% (79/56297)\rUpdating files:   0% (224/56297)\rUpdating files:   0% (303/56297)\rerror: unable to write file <FILE_PATH>
    **^ a few of the above, followed by many many of the following:**
    error: unable to create file <FILE_PATH> No space left on device\n
ERROR: Disk space error - skipping
DEBUG: Unknown res
{
  "res": "disk-space"
}

Then followed by some timing statistics.

Additional context

Similarities to renovatebot/renovate#3449--so I'd waited overnight to make sure that it was consistently failing before filing this issue. The files showing up in the failures are like ~10MB data files; I'm just looking to auto-update a Bazel WORKSPACE in a different part of the codebase, so perhaps there's a way to avoid cloning the whole repo?

Thanks so much! Chris

[rarkins]

By default we use a 4GB SSD for cloning, so anything that's 9GB shallow would have no chance. I don't think we should read too much into the progress indicators in the git log. I have added a server side rule to clone onto an EBS volume with more space instead, so on the next run we should see if it works or not.

I've also added https://github.com/renovatebot/renovate/issues/7840 to discuss how to support partial cloning within Renovate

[rarkins]

I think you should have an onboarding PR now. Repo takes 7+ minutes to clone

[cpsauer]

Confirming that I've gotten the onboarding PR. Thank you so much @rarkins--on a Sunday no less. I really appreciate it, and sorry about the progress-indicator red herring.

[cpsauer]

@rarkins, I'm getting the same disk space error again. Any chance you could bump me up?

In case handy, Job ID is 316572574

Related: Is there any way to get Renovate to notify/email me when jobs error, so I can catch configuration issues fast? Feels like there'd already be a setting somewhere, but I couldn't find with a quick search.

[rarkins]

@rarkins, I'm getting the same disk space error again. Any chance you could bump me up?

In case handy, Job ID is 316572574

This repo has already been switched from SSD to EBS previously, which should give it something like 10GB or more space. We don't have the ability to use any more than that.

Related: Is there any way to get Renovate to notify/email me when jobs error, so I can catch configuration issues fast? Feels like there'd already be a setting somewhere, but I couldn't find with a quick search.

No, there's not any email capabilities at all actually

[cpsauer]

Huh! Things now seems to be back working--with no changes on either of our ends--after just that single, isolated disk space failure.

I had assumed we'd slowly crept up over some maximum size, and the failures were going to persist. I couldn't be more delighted to have been wrong! My new assumption from what you wrote is that EBS should just elastically expand (as the name would suggest, though I thought there might have been an adjustable limit or something) but that there was some other cause for the error?

Anyway, it's been tremendously useful to have Renovate on this repo. Thank you so much, @rarkins, for everything.

[rarkins]

Regardless of AWS naming, the disk size is statically provisioned when the EC2 machine is created 🤷

If it happens with any regularity, please create a new app support discussion in the main repo as maybe we have some residual file system use sometimes (eg too many Docker images downloaded)