[rush] Generate a change file for projects that don't publish via npm

[factoidforrest]

We are using Rush to build a microservice cluster. Some rush packages will deploy to NPM, but after that some others(which depend on the above packages) will be containerized and go to AWS. I'm guessing this is a pretty normal pattern.

It would be really great to use Rush to check what microservices have changed and just deploy those(if they have dockerfiles).

There might be some clever use of version policy to achieve this. Looking for your recommendation.

In Lerna I can do lerna ls --since --json to see what I need to redeploy.

[octogonz]

Is the idea that certain packages are released by running npm publish, but certain other packages are released via some custom script such as some-docker-tool deploy-this-folder?

In other words you want to generalize rush publish to support other release types besides NPM?

@iclanton and are really wanting to do a sprint to improve the publishing/versioning features. We could try to incorporate this requirement into the design. We already use npm pack instead of npm publish in our own repos.

[factoidforrest]

Yes that sums it up very well. Getting all the updated packages in a parseable list is probably enough.

In lerna our pattern was:

1. lerna ls --since --json and store the list of updated packages
2. lerna publish to take all the packages not marked as private:true to npm repository. Our microservices were all private.
3. Then we looped through all the changed packages from step one, checking for dockerfiles and then deploying them all in one parallel job.

When I saw the rush change files, I thought "this is perfect, I can just parse these". But then I realized change files are only generated if shouldPublish is true. Otherwise the change generator never gets triggered and versions never bump.

Could we do something tricky like have a version policy called npm and a policy called container and then publish them differently in two different publish commands?

[octogonz]

What does --since do exactly? I couldn't find any documentation for it.

Could we do something tricky like have a version policy called npm and a policy called container and then publish them differently in two different publish commands?

At the point where Rush calls npm publish, we could just provide a way to customize what command it runs. It could be a version policy option (that affects all packages in the group) or we could provide project-specific options (if you really needed that).

We already encountered this problem ourselves. Someone added _updateApiFile into Rush one time when they needed custom logic to run before publishing, and it really doesn't belong there. Also the _npmPack() felt a little out of place to me as well.

It would be great to provide a general mechanisn for customizing this.

[factoidforrest]

That's good to know it's a common need! I figure a lot of people would like to use rush this way.

--since isn't super well documented. Without an argument, it filters list/ls to just give the packages that have changed since their last tag. https://www.npmjs.com/package/@lerna/filter-options Pretty much perfect for figuring out what to deploy.

Running a custom command is probably good enough. What we really need is a list of all the packages that have changed, because our deployment system can take a list of containers to deploy in parallel. We could have the script command write >> its package name to a file and then read that or something.

If you're thinking about how it SHOULD work ideally, my first thought was adding a shouldDeploy boolean into the rush project config, and then I could create changeFiles and ask rush whatShouldIdeploy but I'm new to rush so that was just my first thought.

I will play with version policies a little more and see if I can do what's needed right of the box.

[octogonz]

--since isn't super well documented. Without an argument, it filters list/ls to just give the packages that have changed since their last tag. https://www.npmjs.com/package/@lerna/filter-options Pretty much perfect for figuring out what to deploy.

Is that because deploying creates a Git tag? Sorry if these are dumb questions -- I really should spend more time reading about Lerna. :-)

I will play with version policies a little more and see if I can do what's needed right of the box.

Version policies are a much better approach to this. I apologize that their current design is fairly confusing (since the requirements were constantly evolving for the people who implemented that portion of Rush). We really need to do another iteration to make it more understandable.

[factoidforrest]

I believe git tag is how it works by default. Deploying does create a git tag for the whole repo, creating an easy way for Lerna to check for changes since the last release(I think, haven't read all the source).

I'm glad you said that because I am getting pretty confused by version policy. It seems like it is one flag trying to do a lot. I'm trying to use it as something I would call 'publishing group'. Maybe it could just use a dedicated doc page that really explains it.

On Thu, Oct 18, 2018, 16:08 Pete Gonzalez notifications@github.com wrote:

--since isn't super well documented. Without an argument, it filters list/ ls to just give the packages that have changed since their last tag. https://www.npmjs.com/package/@lerna/filter-options Pretty much perfect for figuring out what to deploy.

Is that because deploying creates a Git tag? Sorry if these are dumb questions -- I really should spend more time reading about Lerna. :-)

I will play with version policies a little more and see if I can do what's needed right of the box.

Version policies are a much better approach to this. I apologize that their current design is fairly confusing (since the requirements were constantly evolving for the people who implemented that portion of Rush). We really need to do another iteration to make it more understandable.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Microsoft/web-build-tools/issues/884#issuecomment-431143207, or mute the thread https://github.com/notifications/unsubscribe-auth/AAjhEqnc-X6bGMZdj3ufp-8W53IggUaPks5umN-sgaJpZM4XlHiN .

[factoidforrest]

I tried using pack prior to the containerization step, but I noticed it doesn't include any kind of yarn.lock(or package-lock.json) file. Since that file lives in common/temp, that file won't end up in the published packages.

Don't I need a lockfile in each package to be safe with my dependencies?

[octogonz]

So the node_modules folder is not included in your docker container? Instead you run pnpm install as a postprocess after you boot up the container?

With PNPM's model, I believe it would be fairly straightforward to distill a project-specific shrinkwrap.yaml file for a given project. @zkochan does your "shared shrinkwrap file" feature help with this at all?

[factoidforrest]

We just figured out we can use

cp -RLf node_modules local_node_modules && rm -rf node_modules && mv local_node_modules node_modules

To convert rush's symlinks into real files before containerization. It's great because it avoids another install, respects the global package lock, and will also containerize any local packages that are linked but not in NPM.

It would be trivial for rush to someday have a command rush isolate that did this. :)

[octogonz]

@jbcpollak (who was also interested in Docker container deployment) FYI

It would be trivial for rush to someday have a command rush isolate that did this. :)

Today you could probably implement this as a Rush custom command.

Eventually we do want to generalize Rush's publishing scenario to support other release types besides npm publish, and it seems like Docker containers should probably be first class citizens.

[zkochan]

With PNPM's model, I believe it would be fairly straightforward to distill a project-specific shrinkwrap.yaml file for a given project. @zkochan does your "shared shrinkwrap file" feature help with this at all?

A couple of days ago @hulkish was asking about this feature for the same pupose in our gitter chat

Indeed, it would be easy to implement a pnpm command for fetching a dedicated shrinkwrap from shared shrinkwrap. We have that logic already in the pnpm-shrinkwrap package

[ledniy]

@light24bulbs Hi, I guess I have the same problem as you, did you solve the problem with a list of changes? I'm still confusing with version policy and don't understand how to use it properly for our needs.

[factoidforrest]

@ledniy We never got as far as achieving this with version policy before switching to Lerna(for many other reasons). In Lerna just doing ls --since --all --json achieves this. As far as I got was @pgonzal saying to use version policy, but I'm still not sure how to actually set it up properly for this use case of showing changed packages. Basically we are in the same boat.

Version policies are a much better approach to this. I apologize that their current design is fairly confusing (since the requirements were constantly evolving for the people who implemented that portion of Rush). We really need to do another iteration to make it more understandable.

[RichiCoder1]

Not to necro an old thread, but was there an answer for this? I'd like to publish https://aws.github.io/jsii/ projects. So I only want published changed workspaces, but when I publish I want to take over the publish process to jsii-pacmak to actually create the packages and manually publish.

[mathiasmaerker]

I would love to have a publish hook as well, I have a monorepo with very different stuff going on, e.g. publish to npm, publish to docker, do some terraform IaC etc. would be nice to have this feature.

//edit

the creation of changelogs without the need to publish it (or at least to npm) would be absolutly awsome, I already tried to prevent publishing with no success yet

[witcher112]

I just came across this issue, are there any news / new solutions that resolve the problem described here?