Tell renovate to use the ng command #4256

Closed mfechner closed 3 years ago

Which Renovate are you using? CLI, App, or Pro

CLI

Which platform are you using? GitHub, GitLab, Bitbucket Azure DevOps

Selfhosted Gitlab

What is your question?

I started to look into renovate and it is really a very helpful tool. But I have some problems. My projects are normally setup with @nrwl/nx and @angular. To update them correctly you normally execute ng update @nrwl/workspace normally defined in the package.json with the command update. Or for angular ng update @angular/cli @angular/core.

These commands also take care if more files (like package.json, package-lock.json) have to be modified.

For example the last @nrwl/workspace update added a new script "nx": "nx".

But I saw that renovate does only update the package.json which will later cause maybe problems.

How do I have to configure renovate that it correctly uses ng to make these kind of upgrade/mergerequests?

Thanks a lot!

@mfechner I've transferred this to the main Renovate repository to treat as a feature request. Is it possible for you to create a public minimal reproduction repository that I can use to test again? i.e. a baseline master branch that needs an update that is only satisfied fully by running "ng" instead of Renovate's traditional approach?

@rarkins I quickly created a public repo with some files I could reproduce it on my local env. As I use it on my private gitlab server, I'm not sure if renovate bot will create a Merge Request to update NRWL, lets wait.

Repo is here: https://github.com/mfechner/nrwl-renovate

Using npm run update will produce a diff that also modifies the script section in the package.json file:

The renovate bot I have running on my gitlab was not doing this:

Thanks. Am I understanding correctly that you ran ng update @nrwl/workspace but it actually updated all @nrwl/* packages?

Generally we prefer not to use third party CLI tools (e.g. npm or bundler) to update package files directly as usually it doesn't give us enough control (e.g. separating patch/minor/major updates if desired). But in this case it seems we should be using ng update.

@alan-agius4 is this something you're an expert on, or is there anyone on your team who could help advise about the best way for Renovate to integrate with Angular's ng update?

Hi @rarkins, so, ng update is bit different than npm update etc..

Here are some main differences;

node_modules need to be installed prior of running ng update, or at least a global version of the @angular/cli be present.
On top of changing the package version in package.json it run code transformations (if applicable for your version and the package that you want to update offers it), typically these migrations consists of replacing old API usages with new once, update the project structure and add new settings to the workspace configuration file.
the biggest benefits of ng update are during major versions. Example when updating from Angular 8 to Angular 9. During minor and patch release typically there is no need for ng update, as migrations almost never happen in non major releases. (That said, maybe NRWL or other 3rd parties are doing this a bit different and they do migrations in minor and patches)

//cc @clydin who is the real expert in ng update.

If you'd like to have a chat around this, I am definitely available :)

The ideal for Renovate - i.e. fitting in with our existing models - would be if we can run ng update after we update the package.json and package-lock.json. But I assume that may "fool" ng into thinking it's already updated and not doing its magic.

But some good news: Renovate now runs on a shallow clone of each repo so it's entirely possible to do code transformations and commit the results along with the basic package file updates.

This is how we do it today:

Patch package.json manually one or more times to make each update
Run npm install to sync the package-lock.json

Running ng update after those would be simplest, but it should also be feasible to run it instead of the npm install step, if that's possible. To completely replace the process with ng update would be more challenging.

I'm also wondering:

If I run ng update x will it be limited to package x or a possibility it will update others at the same time?
Can I specify an exact version to update to, e.g. ng update x@1.0.1 ?

Hi @rarkins,

If I run ng update x will it be limited to package x or a possibility it will update others at the same time?

Yes, in most cases this will be the case, example running ng update @angular/core will update several other dependencies such as @angular/common, @angular/compiler, rxjs, typescript etc..

Also in some cases to have a green build, users might need to run ng update on several packages at once. Example: ng update @angular/cli @angular/core.

Can I specify an exact version to update to, e.g. ng update x@1.0.1 ?

Yes that is allowed, however If you have already update the package.json and installed the dependencies you can opt to run the migrations only example:

ng update @angular/core --from 7.0.1 --migrate-only

More info about the ng update command can be found here: https://angular.io/cli/update

That being said, at this point in time we don't recommend integrating the ng update into renovate, The main reason being is that @schematics/update package is still not stable and there are some known issues which ideally are solved prior to any integration. There is also a number of questions, such as what will happen if a migration requires user input?

As mentioned previously code migrations typically happens on majors which from our end happen every 6 months, during non major updates renovate can still be used to to update patches and minors if you one packageRules.

Seeing these two PRs https://github.com/mfechner/nrwl-renovate/pull/9 https://github.com/mfechner/nrwl-renovate/pull/2/files

I'd say that the renovate updates are happening correctly.

Thanks @alan-agius4 for the feedback. Based on that I think we probably won't rush into ng integration until you/the team think it's the right time. I'll leave this issue open to capture any new requirements, discussions, or +1's.

It would be nice, if we can only run the migration for specific packages, eg with a packageRule.

{
  "packageNames": ["@nrwl/workspace"],
  "postUpdateOptions": "ngUpdate"
}

and ngUpdate command calls yarn ng update @nrwl/workspace --from 'old-version' --migrate-only. Maybe we can call ./node_modules/ng instead.

@rarkins Do you think that is possible, I use angular too.

When we run npm dedupe as a result of postUpdateOptions we already have some logic to trigger Renovate to do a full npm install prior rather than simply updating the lock file only. In this case we probably need to do the same if we are to run ng update.

This also will need some augmentation of logic, and ideally it's not hardcoded npm hack in lib/workers. The augmentation is that each branch can contain multiple dependency updates, and we usually run npm install and npm dedupe only once per branch. In this case we'd need to detect exactly which upgrades have ngUpdate enabled and run them one by one. I suspect we can do it because we can pass a config.upgrades through to artifacts if not already.

Maybe we should first do some small refactorings to support post-update actions for more managers.

so i would prefer to lift https://github.com/renovatebot/renovate/blob/b0d1de8e595b482cc11a6275ba1ab47ab989e3d2/lib/workers/branch/index.js#L6 to manager level and add optional getAdditionalFiles to specific manager (currently npm only)

then we simply call that https://github.com/renovatebot/renovate/blob/b0d1de8e595b482cc11a6275ba1ab47ab989e3d2/lib/manager/npm/post-update/index.ts#L301

After lockfile updates we can iterate all updates and check for ngUpdate. If found call ng update as described at my comment

I've digged a litle into the code and saw, that we try to update the lockfile without installing node_modules, so we can't rely to have a full install.

I would implement it like lerna:

fetch @angular/cli version from package.json
install cli to ~/.npm/ng@${version}
run ~/.npm/ng/bin/ng update ... for each ngUpdate packageFile / update

Check out how we do a full install if we are doing dedupe. We can do the same here

Hey,

is there anything I can do to help?

ng update is the most reliable way for updating things in the Angular world.

E.g. if I would use renovate now and I would optimistically update typescript and Angular to the latest version there's a good chance that this won't work.

ng update on the other hand is pretty smart and updates things like typescript to a version that it actually supports. I'm not familiar with the CLIs of other frameworks but this lowers the maintenance effort of a project a lot

@kroeder current status is as per @alan-agius4's comment about a year ago:

That being said, at this point in time we don't recommend integrating the ng update into renovate, The main reason being is that @schematics/update package is still not stable and there are some known issues which ideally are solved prior to any integration. There is also a number of questions, such as what will happen if a migration requires user input?

Hello, this is now possible for self-hosted users thanks to @IKatsuba's PR #7632. The documentation there includes the ng example.

renovatebot / renovate