Open moward opened 5 years ago
This is an important feature that we've been wanting to build for a long time.
You already can simulate it to some extent today. Suppose we have:
repo-a/rush.json
"projects": [
{
"packageName": "lib1",
"projectFolder": "libraries/lib1"
},
{
"packageName": "lib2",
"projectFolder": "libraries/lib2"
}
]
repo-b/rush.json
"projects": [
{
"packageName": "lib3",
"projectFolder": "libraries/lib3"
},
{
"packageName": "my-app",
"projectFolder": "applications/my-app"
}
]
And then suppose you created this folder:
bridge/rush.json
"projects": [
{
"packageName": "lib1",
"projectFolder": "../repo-a/libraries/lib1"
},
{
"packageName": "lib2",
"projectFolder": "../repo-a/libraries/lib2"
},
{
"packageName": "lib3",
"projectFolder": "../repo-b/libraries/lib3"
},
{
"packageName": "my-app",
"projectFolder": "../repo-b/applications/my-app"
}
]
When you run rush update
in this folder, it should install the union of the dependencies from both repos in a new bridge/common/temp and then link everything together as expected. On a whiteboard, it's essentially a correct solution. But with one caveat: The two repos need to have consistent versions. (If you're familiar with the Evergreen and XStitch internal tools that coordinate dependency upgrades across repos: In order for a rush bridge
repo to build, the Evergreen/XStitch branch cannot be in a broken state. Otherwise when you try to use rush bridge
to test a fix, instead you'll find yourself involved in work that is equivalent to an Evergreen/XStitch fixup effort.)
So the theory is simple and straightforward. In practice, there are a bunch of loose ends that need to be tied up however (which is why I've tagged this issue as "needs design"):
rush bridge
always need to do the equivalent of rush update --full
(and whatever work is associated with that)Of course we don't need to sort out all these issues on day one. If someone could throw together a quick prototype of a rush bridge
command, we can mark it as "EXPERIMENTAL" and iterate on it. For simple use cases, it may already be better than the manual copying/symlinking workarounds that people use today.
@ThomasMichon who was also interested in this
I've been thinking about this idea for a while, and I've thought about the general requirements one would want from a usage perspective.
These days, one opens a VS Code window into the root folder for a rush-enabled repo, and can use the integrated terminal to open the console. Ideally, you can cd
around there and run npm run ___
or rush ___
and they should 'just work'.
Suppose you have two repos:
src/git/repo-a/
packages/project-1/package.json
packages/project-2/package.json
rush.json
/src/git/repo-b/
packages/project-3/package.json
packages/project-4/package.json
rush.json
I would expect a flow something like this:
~/> cd /src/git/repo-a
/src/git/repo-a> rush bridge add
Added 'repo-a' to the default bridge.
/src/git/repo-a> cd /src/git/repo-b
/src/git/repo-b> rush bridge add
Added 'repo-b' to the default bridge.
/src/git/repo-b> rush install
This repository is part of the default bridge. To remove this repository, use 'rush bridge remove'.
Initializing pnpm
...yada yada
/src/git/repo-b> rush build
This repository is part of the default bridge. To remove this repository, use 'rush bridge remove'.
Building
[project 1]
[project 2]
[project 3]
[project 4]
Done
/src/git/repo-b> rush bridge remove
Removed 'repo-b' from the default bridge.
/src/git/repo-b> cd /src/git/repo-a
/src/git/repo-a> rush bridge remove
Removed 'repo-a' from the default bridge.
If you let users work from their repo's terminal, then you can make most rush script commands work across the whole bridge. Essentially, they all proxy over to the rush configuration for the bridge, which orchestrates the whole thing.
Now, regarding common-versions.json
and other repo-specific configuration, I think the way to get this feature off the ground would be to start with ignoring those features, and see what is required first to make cross-repo linking and command running work.
One thing we could look into doing is making the phantom common/temp
project set up by rush actually 'nest' the bridged repositories as their own projects.
~/temp/rush/bridges/default/
common/temp/
projects/
package-1/package.json
package-2/package.json
package-3/package.json
package-4/package.json
repos/
repo-a/package.json
repo-b/package.json
package.json
When you build the 'root' package.json
file, you make it reference only the sub-repo projects, but also and common versions extractable from all the cross-repo packages.
But for anything common within only a single repo, put it in the package.json
file for that repo, and do the same with entries from common-versions.json
.
I think this would handle that issue gracefully, at least with respect to not breaking each of the bridged repos.
However, ultimately we'd want to 'choose a winner' when it comes to packages that should be common across both repos, so there might need to be some magic involved in produced an aggregate common-versions.json
. I was thinking that maybe common-versions.json
needs support to declare that specific packages do support wildcards, but only to to resolve discrepancies with another whole repo.
For pnpmfile.js
, we can either blindly compose multiple pnpmfile.js
values together, or work with the pnpm
owner to allow the overrides to be scoped when imported under specific sub-trees.
Let's start by designing the command-line experience. Sounds like above you're proposing these new commands:
rush bridge add
rush bridge remove
Some feedback:
The overall user experience seems great to me.
I'd want a way to name the global "bridges", so I can have two bridges active side-by-side
I don't think ts-command-line today lets you specify "add" without being --add
or maybe --add <name>
. Maybe we can work around this. Alternatively, I've been planning to rewrite the parser in ts-command-line to make it more flexible. (@fearthecowboy was asking about this recently on Gitter).
Maybe "join" and "unjoin" are better verbs?
"unjoin" maybe should eject all the repos from the bridge
Rush already puts other temporary files in ~/.rush
. However, if you're setting up a node_modules folder that we're going to symlink against, on Windows it needs to be on the same hard drive. (You cannot create directory junctions between drives, and ordinary symlinks require administrative persmissions to create.) So the bridge definitions can go in ~/.rush
but the common installation should go maybe in the first repo folder that joins the bridge.
Setting up common/temp and common/config under that folder seems reasonable
Regarding common-versions.json, @ThomasMichon and I chatted and we think there should be an @rush-temp/
tarball representing each repo. This would allow each repo's preferred versions to be maintained separately, while still performing a global install for everything.
@octogonz has this feature request progressed any further than what's detailed in this issue? If not, are there any changes in Rush over the past couple years that you think would make this less feasible?
We are running into the same issue described by @moward in our current use of Rush. We have a main repository (iModel.js) that contains platform/base level packages and then several additional repos containing various packages and apps built on top of iModel.js. This has worked well so far but has been a continuous nightmare when making changes across the repos or down in the platform that need to propagate up (sometimes 3-4 different rush repos).
With the new "useWorkspaces" setting in rush.json, I feel like this problem has maybe gotten easier. PNPM will do the hard work of linking folders, and settings like common-versions.json are less important.
I think the correct move would be to update Rush to automatically add those "bridged" packages inside the pnpm-workspace.yaml
file, and then run the installation from there (assuming Workspaces allows for something like that), and that should be it?
Moreover, instead of bridging two entire monorepos together (as suggested above), I think it would be simpler on the user's end to just bridge together specific projects (if you want to make changes to both monorepos at the same time like rush add
, why are they not already merged into the same monorepo?).
My typical usage is I'm relying on a package from another monorepo I already have cloned locally, and don't want to either rsync
or have to publish / update every single time I want to send a change over.
Speaking to the mismatched version detection above, is that something Rush facilitates, or does PNPM facilitate that? If Rush is the one facilitating it, then it should be easy to workaround by just using pnpm-workspace.yaml
. You just run the risk of having different versions of common dependencies, but, you would run the risk of that anyway by just depending on the package normally.
My earlier comment, on my very hackish immediate workaround:
So I'm trying something which I feel might be a reasonable (albeit very hacky) solution if you quick-and-dirty want to get packages linked together.
I've noticed that even though it isn't technically allowed to use link:
version specifiers for dependencies, PNPM seems to play well with it. So I've basically modified my pnpmfile.js
to read the links from a local configuration file, and dynamically update the dependencies of packages in the workspace.
Here's the pnpmfile:
'use strict'
const path = require('path')
const fs = require('fs')
let linkedPackages = {}
const LINKS = path.join(__dirname, '../config/links.json')
if (fs.existsSync(LINKS)) {
const contents = fs.readFileSync(LINKS).toString()
linkedPackages = JSON.parse(contents)
}
module.exports = {
hooks: {
readPackage,
afterAllResolved,
},
}
function afterAllResolved(lockFile, context) {
for (const importerKey of Object.keys(lockFile.importers)) {
// Check to make sure it starts with ../../ because that means it's probably
// inside the current project.
if (importerKey.startsWith('../../')) {
for (const root of Object.keys(linkedPackages.roots || {})) {
for (const linkedPackageName of Object.keys(linkedPackages.roots[root] || {})) {
const finalPath = path.join('../..', root, linkedPackages.roots[root][linkedPackageName])
context.log(`(lockfile) ${linkedPackageName} → link:${finalPath}`)
lockFile.importers[importerKey].dependencies[linkedPackageName] = `link:${finalPath}`
}
}
}
}
return lockFile
}
const DEP_KEYS = ['dependencies', 'devDependencies']
function readPackage(packageJson, context) {
for (const depKey of DEP_KEYS) {
for (const root of Object.keys(linkedPackages.roots || {})) {
for (const linkedPackageName of Object.keys(linkedPackages.roots[root] || {})) {
if (packageJson[depKey][linkedPackageName]) {
const finalPath = path.join('../..', root, linkedPackages.roots[root][linkedPackageName])
context.log(`${linkedPackageName} → link:${finalPath}`)
packageJson[depKey][linkedPackageName] = `link:${finalPath}`
}
}
}
}
return packageJson
}
And here's what the configuration links.json
(inside common/config
might look like:
{
"roots": {
"../other-project": {
"@other-project/package-name": "packages/package-name"
}
}
}
That seems to work in that it actually links the dependencies to the proper locations, but I'll note a few things:
rush update
- the only side-effect of this is it looks like it can't resolve the package versions properly, and therefore doesn't update the shrinkwrap file. This means package updates are not calculated properly (and it actually looks like it might be impacting my TypeScript compilation if I'm understanding this properly, see below).Not sure this is the best approach to resolving this issue, but I do like the idea of defining all of the links in a single file and it applies to all packages in the monorepo instead of having to run pnpm link
for each and every project.
UPDATE 09 / 24 / 20 - It looks like the initial implementation I had (the piece above minus afterAllResolved
), didn't jive too well with Rush. So after some fiddling / looking through the code, I think I have it working reasonably well, although all bets are off as to what happens when you try to publish with active links installed (I would assume, nothing?)
I made the following changes to the Rush codebase to get this to work:
apps/rush-lib/.../PnpmProjectDependencyManifest.ts
if (!shrinkwrapEntry) {
// We don't support linked dependencies inside the shrinkwrap file.
if (throwIfShrinkwrapEntryMissing && !version.startsWith('link:')) {
throw new InternalError(`Unable to find dependency ${name} with version ${version} in shrinkwrap.`);
}
return;
}
apps/rush-lib/.../WorkspaceInstallManager.ts - inside _createPerProjectManifestAsync()
, around line 567.
// We want to skip linked versions, as we don't have an automatically-computed integrity for them.
if (version.startsWith('link:')) {
continue;
}
apps/rush-lib/.../PnpmShrinkwrapFile.ts - inside _parsePnpmDependencyKey
, around line 624
// We don't support links in shrinkwraps.
if (!result && !pnpmDependencyKey.startsWith('link:')) {
throw new Error(
`Cannot parse PNPM shrinkwrap version specifier: "${pnpmDependencyKey}"` +
` for "${dependencyName}"`
);
}
My understanding is this part of the codebase parses the PNPM lockfile generated inside the temp
directory, and pulls out the integrity fields, storing those in individual shrinkwrap files for each project, and that in turn has something to do with not re-installing all of the dependencies each time you run rush update
.
And of course, this doesn't work if you tamper with the versions inside your pnpmfile.js
(although the common-versions.json
file does the same thing after inspecting its pnpmfile implementation, so I'm not sure if this error comes up or not when using that).
Alternatively, I'm assuming you could get around the code change by disabling the legacyIncrementalBuildDependencyDetection
inside experiments.json.
It would be interesting to see if Rush would be interested in supporting that links.json
structure; the purpose being you define all of your "external-package" links, and it works with PNPM workspaces to install those links properly.
As for how this would work using potential future workspaces or being integrated into the core linking functionality, I have no idea.
If this is interesting, I could start working on a PR with a new configuration file for defining these and an implementation.
As discussed on an internal wbt thread, I'd like to propose a
rush bridge
. The purpose would be to connect two or more Rush repos together so that:To use this tool, the user would create another
rush.json
file that points to several Rush repositories. This tool would then handle installing and symlinking dependencies and building as if all the known packages were in one Rush repo.Some context: My team works primarily in one Rush repository, but we have dependencies on packages from other Rush repositories and are furthermore consumed by a different downstream repository. Right now, there isn't a great way to test how changes in one package affect downstream consumers. Our current options are:
npm and pnpm, on the other hand, have support for local filesystem dependencies in standalone repos.
This would greatly help to improve our development, testing, and automation work.