Make it possible to validate Dependabot config before it lands on `main`

[saramah]

There used to be a dependabot config validator at https://dependabot.com/docs/config-file/validator/, but it has gone missing. Is there a replacement?

references: https://stackoverflow.com/questions/63943201/how-do-i-test-dependabot-before-merging-config https://github.community/t/how-to-validate-dependabot-config-file/218636

[mark-stopka]

Same issue here...

[jurre]

The Dependabot app registers a "CheckRun" in GH whenever the config file changes: https://github.com/dependabot/dependabot-core/runs/4327679442

Does that help?

[hfhbd]

@jurre Not really, as the CheckRun only runs at the main branch, not in a PR. This is even visible in your linked example: https://github.com/dependabot/dependabot-core/pull/4454/checks, in the checks there is no validator.

See also: https://github.com/dependabot/dependabot-core/issues/3878

[jurre]

@hfhbd ah yes, that's still not working :( I'll see about getting that prioritized.

[jurre]

In an effort to be transparent, I just investigated this a little and it seems the issue that prevents the check from running on PRs is fairly deep inside GitHub itself, and will require us to rejig some things internally so unfortunately not an easy fix. Will still try to get it prioritized either way, but just might take a little longer for us to roll out a fix because it involves changing a bunch of moving parts :(

[brrygrdn]

👋🏻 Just removing the needs-triage as we've been looking at this internally. There is some work we want to do around Dependabot configuration and enablement, so I've made a note of this problem as part of that. I don't have anything to share right now, but this is something we'd like to do better on.

[jlosito]

@jurre @brrygrdn I understand that some additional rejigging would need to be done, but would there be the possibility to release some sort of cli tool which people could run in order to validate the configuration file? Something such as dp-lint /path/to/.github/dependabot.yml.

[jeffwidman]

I realize the original post title was asking for a stand-alone validator, but I suspect the real intent here is to make it possible to validate the config before it lands on master/main.

That could be implemented via a standalone validator, but as discussed later in this issue's comments, it could also be via a check that runs on the PR before merge.

So I reworded the title to focus on the problem, rather than describing only one possible fix implementation.

And yes, we do very much want to resolve the hurdles mentioned in https://github.com/dependabot/dependabot-core/issues/4605#issuecomment-1023250637 and get this resolved.

[jeffwidman]

I just noticed that the GH action that validates dependabot.yaml just ran on a PR: https://github.com/dependabot/dependabot-core/pull/5513/checks?check_run_id=8648478306

I'm a bit surprised because AFAIK we didn't make the the internal changes described in https://github.com/dependabot/dependabot-core/issues/4605#issuecomment-1023250637.

The only thing that is different from a normal PR is that I updated the PR with a rebase off main... so perhaps that triggered it? 😕

[jeffwidman]

I just noticed the check also ran on this PR branch: https://github.com/dependabot/cli/pull/17/checks?check_run_id=8862312262

And that PR doesn't touch dependabot.yml file... BUT, interestingly, this PR that I merged 15 mins before does change the dependabot file: https://github.com/dependabot/cli/pull/18

Given that we haven't made internal changes, I suspect this is caused by an unrelated race condition of some sort, but I don't have time to dig into that right now. So just leaving a breadcrumb if we keep seeing this occasionally and want to investigate.

[jeffwidman]

As a quick update on this issue, this is still on our radar. Some of the internal changes that are needed to unblock the work on this are happening this quarter. Again, that'd only unblock the actual work, but it's a step in the right direction.

[maxmoehl]

I stumbled across this issue because I'm about to add dependabot to a repo and wanted to verify my config. Initially the check didn't show but after I force-pushed the branch once the check suddenly appeared. Don't know if this is reproducible: cloudfoundry/haproxy-boshrelease#367

[MuhammedMahrous]

Any updates on this?

[wesley-harding]

This is currently hindering my adoption of Dependabot. It's a huge pain to have to go through our entire SDLC just to learn a validation failed.

Further complicating this is the docs are, well, rather underwhelming. Docs give direction indication, but there's a lot left to imagination.

[larouxn]

👋 I'm not sure exactly how this GitHub Action is run/added to our merge flow but it does appear possible to validate dependabot.yml changes prior to merging to main. In the following case I mispelled package-ecosystem.

Then corrected it.

[nickzelei]

I ran into this myself after merging a bad config into main (which was previously valid somehow). I put up another PR to try and fix it and suddenly the same job that didn't run on my previous PR is now running to validate the dependabot config.

Some weird behavior here. Why didn't it run previously on the PR that ended up breaking, but now it runs on the subsequent PR to fix it?

[bugron]

I'm sharing this here in case anyone finds it useful.

If your Dependabot config file is located at .github/dependabot.yml (notice the extension) simply run the following command in the root of your repository. See the docs regarding passing another path.

npx -y @bugron/validate-dependabot-yaml@latest

I also needed to validate a Dependabot config file without committing to the default branch first so I've published a CLI tool @bugron/validate-dependabot-yaml that does exactly that and more. It is capable of logging validation errors both in Markdown and JSON so can naturally used in GH Actions for sticky comments and post-processing errors. Note, the tool isn't perfect, but should do the job.