search

dependabot / dependabot-core

🤖 Dependabot's core logic for creating update PRs.
https://docs.github.com/en/code-security/dependabot
MIT License
4.63k stars 991 forks source link

Improvement Proposal: Configurable Pull Requests #2672

Closed ChrisCates closed 1 year ago

ChrisCates commented 3 years ago

Summary

Dependabot provides several pull requests all for the same repository. These diffs in the repository are pretty trivial and generally involve changes only relating to package managers. Pull Requests needs to be configurable from a dependabot.yml file.

Implementation

Commit Structure

Currently, commits are structured for only a single version change.

@dependabot
Bump elliptic from 6.5.0 to 6.5.3 …
00dff40
Bumps [elliptic](https://github.com/indutny/elliptic) from 6.5.0 to 6.5.3.
- [Release notes](https://github.com/indutny/elliptic/releases)
- [Commits](indutny/elliptic@v6.5.0...v6.5.3)

Instead, add all changes to a single commit and have it structured something like this.

@dependabot - Version Updates

Bump elliptic from 6.5.0 to 6.5.3 … 00dff40
Bumps [elliptic](https://github.com/indutny/elliptic) from 6.5.0 to 6.5.3.
- [Release notes](https://github.com/indutny/elliptic/releases)
- [Commits](indutny/elliptic@v6.5.0...v6.5.3)

Bump lodash from 4.17.15 to 4.17.20 ... 0e4a060
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.20.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](lodash/lodash@4.17.15...4.17.20)

Configuration with dependabot.yml

For complete and full granular control of how @dependabot configures pull requests. A dependabot.yml file seems most ideal.

Example Configuration:

ignore:
  - list
  - packages
  - here

group:
  - list
  - packages
  - here

group:
  - multiple
  - iterations

Example Configuration Group All:

group:
  - *

Example Configuration Grouping Based on organization deps:

group
  - @org/*

ignore

The ignore key in the .yml file ignores all the packages

group

The group key in the .yml file groups all packages into the same pr.

Both keys can be iterated multiple times in the .yml file.

Project Structure

If the project is running on a monorepo. You can have multiple .yml files in each respective folder. @dependabot will walk through each folder to find the .yml file.

fkirc commented 3 years ago

I would prefer to list not only individual packages, but to also have the ability to list entire organizations. For example, organizations like @types or @symfony could be good candidates for an organization-wide grouping. Please see https://github.com/dependabot/dependabot-core/issues/1190#issuecomment-718919532 for further details.

Listing individual packages is fine in some cases, but there are also cases where an organization-wide grouping is easier to maintain than individual packages.

For example, I have multiple dependencies of the @capacitor organization: https://capacitorjs.com/ Whenever Capacitor releases a batch of updates, then I do not only want to merge all of those, but I also need to merge all of those in order to have a consistent Capacitor-setup.

ChrisCates commented 3 years ago

@fkirc, would love to be able to do something like this.

group:
  - @symfony/*
  - @laravel/*
...

I agree with that!

Jacob-Morgan commented 3 years ago

This might be useful as a real world example, we had similar issues using a custom variant of the Dependabot script with our self-hosted gitlab server, developers complained about the number of Merge Requests that where being generated. To fix this we extended the dependabot configuration with group_updates which has similar traits with ignored_updates and allowed_updates, where by you state package names that should be grouped.

version: 1
update_configs:
  - package_manager: dotnet:nuget
    directory: /Source
    update_schedule: daily
    ...
    group_updates:
      - match:
          dependency_name:*

The result of including all packages into a single, so not only do the developers get bothered less with merge messages, the number of auto-merge fails due to conflicts requiring a rebase has been almost reduced to zero. Grouped Packages Updates

h-no commented 3 years ago

I have a monorepo and even though it is not very large at the moment I expect the sheer volume of PR's will be difficult to handle as the repo grows. I'd love to be able to configure dependabot like this. Great idea!

@fkirc, would love to be able to do something like this.

group:
  - @symfony/*
  - @laravel/*
...

I agree with that!

That looks like a good idea for handling grouped dependencies, @ChrisCates why not update your proposal with these examples?

ChrisCates commented 3 years ago

@Jacob-Morgan, thanks for the reference to that fork! Very important.

@h-no, agreed, going to add that!

I still feel the structure of this IP needs to be sorted out. But this is a rough draft so to speak and just something to use as a reference.

davidspek commented 3 years ago

I am currently looking at the adoption of dependabot for the Kubeflow project, and as many others the amount of PRs that will be created for a given repository (and the resulting load on the AWS testing infra) are the man reasons for concern. The ability to group dependency updates together seems to be the most effective way to combat this issue, along with the problem that some dependencies need to be updated together to be functional. To me it seems the solution provided in https://github.com/dependabot/dependabot-core/issues/2672#issuecomment-719130140 is a good way forwards. What would be required to start getting this integrated into dependabot?

halter73 commented 3 years ago

This would also help us out in the https://github.com/dotnet/aspnetcore repo.

mwaddell commented 2 years ago

Solving this is a more general way is a bit beyond dependabot's scope. However, you should be able to solve this with a custom github workflow. You would create an action that runs whenever a new PR is created by dependabot and it uses github-script to locate instances of PRs for related dependencies across your repo and then merges the changes from the new PR into the existing one and closes this new PR with a comment referring to the other PR.

Related to #3856 #1296 and #1190

spenserblack commented 2 years ago

Sorry for commenting on an issue that hasn't had activity for a while, but I think having wildcards could also be useful. For example, Vue CLI dependencies are released together all at once (https://github.com/vuejs/vue-cli/commit/ef08a08c41b028a2484f262414a8c91d151febc7), so it's useful to have one PR for them all.

Something like this, for example:

group:
  - @vue/cli*

I realize this is already very similar to the concept of "grouping all" and "grouping organizations", but I think those concepts can be combined into a more generalized pattern-matching for grouping/ignoring.

If you agree, maybe the proposal can be updated to specify that * is a pattern-matching wildcard in general, and not just for combining all deps or organization deps?

Also, some other things to consider:

pauleustice commented 2 years ago

Hello everyone! We at Legal and General have just open sourced Dependabot Batcher, a GitHub Action that we've been trialling internally for some time. When run on a schedule (or ad-hoc, if you prefer), it will combine all open Dependabot PR changes into one.

The PR provides links to the original PRs, while closing them down. This can save a lot of CI time as well uncluttering your Pull Requests page.

Here's what the generated PR looks like:

We hope that someone finds this useful as we do, and look forward to hearing any feedback you might have!

jurre commented 2 years ago

Interesting @pauleustice, can you share a little about how this deals with conflicts?

pauleustice commented 2 years ago

@jurre Sure. Here's what we have in the readme:

ℹ️ When the Action is unable to merge a Dependabot PR, it will keep the original PR open. This usually happens when there are updates for consecutive dependency entries in your package.json file, which cause merge conflicts. Merging the batched PR will mean it gets picked up on the next run.

You will see the branches that could not be combined in the logs, for example:

🚫 These branches could not be combined: dependabot/npm_and_yarn/nrwl/linter-14.5.1 dependabot/npm_and_yarn/nrwl/eslint-plugin-nx-14.5.1

Of course, you can manually add any merge conflicts to the combined PR if you wish. In our team we just merge the batched PR and then they get picked up on the next run.

spenserblack commented 1 year ago

:tada: https://github.com/github/combine-prs

jeffwidman commented 1 year ago

Thanks @spenserblack I hadn't seen that. :grin

Hopefully it's a helpful workaround until we get a chance to actually get this solved upstream here in :dependabot:.

Looking through the comments, this is pretty clearly a duplicate of a combination of:

Also related: