Major release backwards compatibility for hosted users

[rarkins]

What would you like Renovate to be able to do?

Provide a way to make backwards-incompatible ("breaking") changes to Renovation functionality but maintain old behavior for existing orgs and/or repos.

For example if Renovate switches from enabling or disabling a feature to the opposite - even if it's a good idea - it ideally should have the capability to maintain existing behavior for existing users.

If you have any ideas on how this should be implemented, please tell us here.

First of all, this setting should ideally be controllable per-org or per-repo. Per-org would mean that even new repos within the same org would keep the "old" behavior. If it's per-repo, it means that only existing repos would keep the old behavior any time there's a change.

There would also need to be the option for individual repos to either:

• Opt out of this concept completely (always give me the new behavior), or
• Opt out of specific migrations where they're happy to adopt the new behavior

One way I'm thinking of implementing this is that we introduce a new category of preset, e.g. legacy:*. If any legacy presets are included, a note should be made in the Dependency Dashboard. The idea is that repos or orgs should:

1. Be aware that legacy presets are being applied, and
2. Be able to retain the legacy behavior or remove it, and either way remove the legacy warning

So then when we have a major upgrade, there could be a legacy preset most times, e.g. legacy:v35. This preset would be a wrapper around 1 or more legacy:* presets, and then each would then contain the actual config to revert default behavior (e.g. prHourlyLimit: 0).

Ideally the Renovate administrator could have granular control over this too. e.g. they can read the release notes, decided that non-zero PR limits are a good thing, and not include the legacy presets for it when upgrading to v35.

The next challenge is statefulness, i.e. renovate knowing which repos to keep backwards behavior and which are new. This could be done via either:

• A stateful backend such as the official Mend Renovate App which has a DB, or
• Approximating it based on the date when the onboarding PR was merged, config file was first committed, or something like that

Is this a feature you are interested in implementing yourself?

Maybe

[viceice]

This is probably not solvable without some statefulnes but sounds interesting.

I don't need that behavior, as i always want latest behavior. 🙃

[secustor]

WDYT about adding a major/schema version tag to renovate.json?

That way we know from which level to migrate and we can offer upgrades to the current version with the old behaviour

Basically there would be then a preset to opt in this config upgrades and the legacy:* presets which actually are doing the migrations.

So the workflow would be

V34 renovate.json:

{
  "version": "v34",
  "presets": [
     "keepBehaviorOnMajorUpgrade"
  ],
  "SomeConfig": "ValueToBeMigrated",
}

PullRequest renovate.json:

{
  "version": "v35",
  "presets": [
     "keepBehaviorOnMajorUpgrade"
  ],
  "SomeConfig": "MigratedValue",
}

If there is no keepBehaviorOnMajorUpgrade preset, we only upgrade the version and open a PR with the changes between the majors.

[rarkins]

@secustor I like this idea too. One challenge would be knowing whether to do it at the repo level or upgrade a shared preset instead

[secustor]

I would simply make this two different presets and each would be separately evaluated.

[rarkins]

We also want a way of migrating the existing user base, e.g. we assume that no version means v34 (or whenever it is we eventually start supporting this)

[secustor]

How a about yet another preset useConfigSchema which is added to config:base? That schema would add the version of the currently running renovate instance to renovate.json.

So everybody which uses our default values will be opted-in everybody with customisation will keep the same behaviour as before. The second batch of users have to check changelogs anyway as we introduce breaking changes and can then decide to opt-in.

[rarkins]

We still need a way to indicate this to the end user in repo (e.g. for the hosted app)

[secustor]

Are you referring to this?

So everybody which uses our default values will be opted-in everybody with customisation will keep the same behaviour as before. The second batch of users have to check changelogs anyway as we introduce breaking changes and can then decide to opt-in.

I would add this as additional Note on the PR which initially adds the version to renovate.json.

[rarkins]

Something I still think is undecided is whether PRs targeting renovate.json for existing users are required. A few more thoughts:

We could make Renovate always embed its current major version into a field like renovateCompatibility in every onboarding PR. This means new users should always get it, unless they manually delete it. Edge case: users who skip having a repo config.

This leaves existing users with no renovateCompatibility. In this case we'd implicitly assume e.g. v35 compatibility (whenever as the last major release without this feature).

Another potential problem is people being confused about which version is run. e.g. they have renovateCompatibility=v35 but actually they're running Renovate v37. They may get confused and think they're running Renovate v35.

[rarkins]

I like the idea of having a compatibility field in renovate.json. We could:

• Assume it is e.g. v35 if unspecified (let's assume we add this feature in v36)
• Add it to each renovate.json during onboarding from then on (i.e. whatever the current Renovate major version is)

We need to think about legacy users - at which point, and how do they start having a compatibility field in their config?

Also for users which would like to centrally control their compatibility, how do we do that? I'm thinking that compatibility could be set:

• By the bot admin, in global config (i.e. don't make individual repos worry about it), or
• As a shared preset (e.g. centrally controlled even if not bot admin), or
• In individual repos

Consider the hosted app. We could add a WARN to each repo which has no compatibility setting, only if there are backwards compatibility settings. e.g. if v36 has no legacy presets then no need to bother people. People could remove the WARN by either:

• Adding compatibility to each repo, or
• Adding it to their shared config, if they have one

We could also support the concept of compatibility=* which means "don't bother me, just give me the latest always".

[HonkingGoose]

Preventing mode confusion

Another potential problem is people being confused about which version is run. e.g. they have renovateCompatibility=v35 but actually they're running Renovate v37. They may get confused and think they're running Renovate v35.

We should tell our users in the Dependency Dashboard and in Renovate's own pull requests:

• The runtime/program version of Renovate that ran
• The "behavior" version that's used

Related issues

• https://github.com/renovatebot/renovate/issues/16100
• https://github.com/renovatebot/renovate/issues/16359

We're thinking of adding a best:practices preset, how will that work with "major release backwards compatibility"? I think the most logical thing is for users who use best:practices to just get the latest major release behavior. I assume that new major releases will always follow our own best practices. 😄

Should we work on getting automatic config migrations going first? That way we can automatically update the compatibility field in renovate.json files.

[rarkins]

It occurred to me that this is a similar concept to the Go mod directive in some ways.

A colleague also alerted me to this concept in Ruby: https://www.fastruby.io/blog/rails/what-does-load-defaults-do.html

[rarkins]

While building this I've realized there's an important point to make: this backwards-compatibility should be about repository config options (such as branch names), and not global/admin options (such as autodiscover). For self-hosted global/admin options, our usual approach of documenting changes in major release notes works fine.

[rarkins]

Design decisions so far:

• Use field name renovateCompatibility
  • even though we have a field called compatibility and there could be some confusion
  • not $compatibility
• The value is an integer (e.g. 38) and not e.g. v38. Easier to validate or is it worth it for readability to use v38?
• The value maps to major renovate releases, i.e. we don't start at v1
• Allow special value of 0 to mean "I always want to use the latest, do not bother me with compatibility"
• Onboarding PR gets a compatibility value inserted unless the bot admin has configured a compatibility of 0 to indicate they don't care
• Even if a bot admin specifies 0, a repo can still specify a version if they care
• We merge the repo config and then resolve any presets, and look for a value of renovateCompatibility. If it's less than the latest, then we apply the compatibility presets to the global config and then reapply the repo config over the top of that. This is so that repo config takes precedence over compatibility config

Still to do:

• Where/how to warn users that they have compatibility settings? e.g. in every PR, and in the Dashboard issue?
• Where/how to document each major release's compatibility settings for easy user consumption?
• What to do if a major release doesn't require any compatibility settings? Do we upgrade everyone's configs unnecessarily, or instead not upgrade them and potentially introduce confusion?
• Are migration PRs necessary for day 1?
• Should migration PRs be included as part of the existing "Config Migration" or be a separate/specific PR just for compatibility?

[HonkingGoose]

• The value is an integer (e.g. 38) and not e.g. v38. Easier to validate or is it worth it for readability to use v38?

An integer is fine. The config option name is more important than the version number in the config itself.

• Where/how to warn users that they have compatibility settings? e.g. in every PR, and in the Dashboard issue?

Warn as often as we can. So in each PR and the Dependency Dashboard issue. I don't know if you can warn in the Mend dashboard?

If possible, I'd like us to link to the source file of the configuration. So that users can see where the behavior is coming from.

Mockup time for the warning in the PR and Dashboard:

### Renovate compatibility version

Renovate's `renovateCompatibility` is set to `38`. The source of this setting is in: [link to source file](). Read the [`renovateCompatibility` docs]() to learn more.

• Where/how to document each major release's compatibility settings for easy user consumption?

How about creating a page similar to the Renovate docs, Release notes for major versions of Renovate, but for renovateCompatibility settings? So you would see:

• Versions sorted newest to oldest
• Breaking changes
• Maintainer commentary
• Tips/tricks/gotchas

• What to do if a major release doesn't require any compatibility settings? Do we upgrade everyone's configs unnecessarily, or instead not upgrade them and potentially introduce confusion?

Upgrade always. Skipping versions seems more confusing to me. The special renovateCompatibility page might say: "renovateCompatibility 39 has no changes."

• Are migration PRs necessary for day 1?

Yes, because then:

• we onboard everybody at once
• all users will know about the new feature
• we can explain how to opt-out in a special pinned issue/discussion, even better if we can link to that issue/discussion, or explain it in the PR that enable the feature somehow
• you can schedule the release of the feature, so you're ready to answer questions about it

• Should migration PRs be included as part of the existing "Config Migration" or be a separate/specific PR just for compatibility?

The config migration for renovateCompatibility should be handled the same as all other config migration PRs. This way there's one concept to remember: "If my Renovate config is outdated, I'll get a PR to fix it, and that PR always looks about the same.".

Splitting things into different config migration PRs can be confusing: "Which PR is for what thing?", "Why are these PRs split into two, they both change my config, so why not have a single PR.".