Renovate jobs not running on some repos.

[arnabkd]

Which Renovate are you using?

WhiteSource Renovate App

Which platform are you using?

GitHub.com

Have you checked the logs? Don't forget to include them if relevant

Yes, there are none for the last few days.

What would you like to do?

Could you look into what can be done apart from removing some repos and adding them back? That seems to have solved the problem for now, but I would like to avoid this in the future.

[rarkins]

Are these your personal repos? i.e. github.com/arnabkd/* ? It's possible that they got classified as "stale", which means they are checked no more than once per week. Next time please let us know the repository name so it can be checked on the backend.

[arnabkd]

Hi!

These repos are owned by my organisation at work github.com/finansnorgeforsikringsdrift/tffauto-declus How does a repository get marked as stale? Can you check if this particular one is marked as stale? There are currently 5 pending PRs on the repository (which is the limit). However, increasing the limit did nothing and the same goes for clicking the checkmark for force-running renovate on the repository.

[rarkins]

Repositories get marked as stale if they've received no commits to the base branches for at least two weeks. finansnorgeforsikringsdrift/tffauto-declus is not marked as stale though. Taking a deeper look, there seems to be a strange problem with the repo where it's either timing out or locking up somehow. I see that its jobs get accepted by workers but they go silent until they get an automatic restart due to inactivity. I'll try to troubleshoot the problem

[rarkins]

Finally had a breakthrough and it seems the problem is it timing out trying to query your nexus server.

[arnabkd]

That’s very strange because previously the nexus server didn’t cause timeouts. Is there a way to tell renovate for ignore all packages that come from our nexus server?

[rarkins]

Yes, we added a way to do that just recently and I'll look to apply that today. However when I added the same unreachable registry to a test pom.xml locally I can't reproduce the problem.. so potentially I made an incorrect assumption.

[arnabkd]

Maybe there are some caching issues? :P

[rarkins]

Does the repo have any Bundler dependencies?

[arnabkd]

If you mean Bundler as in the Ruby package manager, then no.

[rarkins]

FYI this repo was found to be one of three (out of 150k+) that somehow manage to kamakaze our job workers. Soon after starting renovating, the worker goes to essentially 0% CPU and never returns a result until it's eventually detected and killed 1+ hours later. I've disabled these three repos including yours in the app for now and would appreciate it if you can help me debug it.

Unfortunately no approach is a perfect reproduction, but we can try.

The safest way to start could be to fork the troublesome repo it into your personal space and run the Renovate CLI against it, e.g.

• fork it
• delete the existing renovate/* branches
• generate a personal access token for your account
• if you're comfortable with Docker, run it like docker run --rm -e LOG_LEVEL=debug -e RENOVATE_TOKEN=$GITHUB_TOKEN renovate/renovate arnabkd/tffauto-declus > debug.log
• see if it completes or not. if it does, then it seems it's not reproducing the fault. but maybe the logs point to something helpful anyway

[arnabkd]

Hi, I would be happy to help you debug. However, can we coordinate this privately? Maybe if I can reach out to your email and we can coordinate over Teams/Zoom/Slack?

[rarkins]

I'm going to be fairly offline the next week and not responding particularly quickly, but feel free to email the address on my profile.

[arnabkd]

I sent you the debug log to your email.

[henrist]

@rarkins I'm seeing something similar on some our repos, but I suspect it's due to the "stale" classification you mention. Is this documented anywhere so I can understand how it works? Is it correct that a commit/push will retrigger a job or move it out of stale until some time later?

It would be great if the Renovate Dashboard (https://app.renovatebot.com/dashboard) could show if a repo is stale, and also the last time it ran. As logs are only visible for a few days it's a bit difficult to know what is wrong when Renovate has apparently not run for a while.

I've had issues before were Renovate stopped working on some repos (and only reinstall seemed to help), without us knowing, so I've built a way to see when Renovate last updated the Dependency Dashboard issue, however it's causing lots of false positives since some repos don't have any updates. Currently, out of 119 repos set up with Renovate, there are 29 repos where it's more than 40 days since last modification.

Example of such repo: https://github.com/capraconsulting/siren-util

[rarkins]

You should always be able to tick this checkbox in a Dependency Dashboard to trigger a run, even if the repo is stale:

(if not then we'd treat it as a bug)

[henrist]

That's what I typically do to ensure the repos are up-to-date. However it seems to not bring the repo out of stale state (it only performs one run).

Do you have any details on my question regarding how a repo gains stale state, and how it looses it? When can I expect a repo to pick up dependency updates? I have an example of one repo now that hasn't picked up an update that was released 10 days ago - see https://github.com/capralifecycle/fnf-helsenett-http4k-cors (private repo). I've intentionally not triggered this to see how it behaves.

[rarkins]

The app looks for any log message 'latest commit' and extracts the time stamp from that. If it's more than 60 days old then it's classified as stale.

I can confirm that capralifecycle/fnf-helsenett-http4k-cors is marked as stale and had been last checked on April 15.

I ran it manually just now, and this log message was produced:

{"level":20,"branchName":"master","latestCommitDate":"2021-02-13T00:47:25+00:00","msg":"latest commit","time":"2021-04-26T05:11:31.785Z"}

Therefore it was not classified as non-stale.

There are 5 branches listed, with various states. e.g. one was "Skipping branch creation as not within schedule"

[henrist]

Thanks, that's very useful details. How often is a stale repo rechecked? Considering my fact that approx 10 days had already passed without any run.

In this case, since the repo itself does not have any recent commit, the new dependency will not really be updated until long time, since it needs multiple runs to complete (expected schedule, create branch and wait for status, then merge if still up-to-date).

[rarkins]

It's currently a manual process but we'll try to get them checked each weekend from now on.

[henrist]

Aha. That sounds like something inconveniently to rely on, and useful to know about. Is this documented behaviour? Will a retrigger (in the Dependency Issue), detecting a new commit in the repo, move it out of stale, or is it fully managed by your "manual process" so we just have to wait for it to happen?

In https://github.com/renovatebot/app-support/issues/61#issuecomment-656682114 it was mentioned that stale repos was checked no more than once a week, som I'm a bit confused to what is actually happening.

[rarkins]

It's not documented to that detail, and decisions like how many VMs are running or when stale repos are scheduled is considered an internal implementation detail for now. Checking a repo around weekly for updates when it hasn't been updated for months seems reasonable.

If a repo runs, e.g. due to a dashboard trigger, and is no longer stale, then it should automatically get moved out of the stale state and go back to being checked every 1-2 hours.

You're also welcome to self-host, if you find you need more frequently scheduling - the functionality is the same.

[henrist]

Thanks for clearing up on the details! :+1: