microsoft / rushstack

Monorepo for tools developed by the Rush Stack community
https://rushstack.io/
Other
5.81k stars 592 forks source link

[rush] [feature request] rush bridge for linking multiple rush repos #876

Open moward opened 5 years ago

moward commented 5 years ago

As discussed on an internal wbt thread, I'd like to propose a rush bridge. The purpose would be to connect two or more Rush repos together so that:

  1. A package in one Rush repo can depend on a package in another repo
  2. The repos share external package dependencies
  3. The repos can be built together in the correct dependency order

To use this tool, the user would create another rush.json file that points to several Rush repositories. This tool would then handle installing and symlinking dependencies and building as if all the known packages were in one Rush repo.

Some context: My team works primarily in one Rush repository, but we have dependencies on packages from other Rush repositories and are furthermore consumed by a different downstream repository. Right now, there isn't a great way to test how changes in one package affect downstream consumers. Our current options are:

  1. Mess around with the common node_modules, overwriting the installed packages with our changes
  2. Publish new package versions without testing against the packages'

npm and pnpm, on the other hand, have support for local filesystem dependencies in standalone repos.

This would greatly help to improve our development, testing, and automation work.

octogonz commented 5 years ago

This is an important feature that we've been wanting to build for a long time.

You already can simulate it to some extent today. Suppose we have:

repo-a/rush.json

"projects": [
    {
      "packageName": "lib1",
      "projectFolder": "libraries/lib1"
    },
    {
      "packageName": "lib2",
      "projectFolder": "libraries/lib2"
    }
]

repo-b/rush.json

"projects": [
    {
      "packageName": "lib3",
      "projectFolder": "libraries/lib3"
    },
    {
      "packageName": "my-app",
      "projectFolder": "applications/my-app"
    }
]

And then suppose you created this folder:

bridge/rush.json

"projects": [
    {
      "packageName": "lib1",
      "projectFolder": "../repo-a/libraries/lib1"
    },
    {
      "packageName": "lib2",
      "projectFolder": "../repo-a/libraries/lib2"
    },
    {
      "packageName": "lib3",
      "projectFolder": "../repo-b/libraries/lib3"
    },
    {
      "packageName": "my-app",
      "projectFolder": "../repo-b/applications/my-app"
    }
]

When you run rush update in this folder, it should install the union of the dependencies from both repos in a new bridge/common/temp and then link everything together as expected. On a whiteboard, it's essentially a correct solution. But with one caveat: The two repos need to have consistent versions. (If you're familiar with the Evergreen and XStitch internal tools that coordinate dependency upgrades across repos: In order for a rush bridge repo to build, the Evergreen/XStitch branch cannot be in a broken state. Otherwise when you try to use rush bridge to test a fix, instead you'll find yourself involved in work that is equivalent to an Evergreen/XStitch fixup effort.)

So the theory is simple and straightforward. In practice, there are a bunch of loose ends that need to be tied up however (which is why I've tagged this issue as "needs design"):

Of course we don't need to sort out all these issues on day one. If someone could throw together a quick prototype of a rush bridge command, we can mark it as "EXPERIMENTAL" and iterate on it. For simple use cases, it may already be better than the manual copying/symlinking workarounds that people use today.

octogonz commented 5 years ago

@ThomasMichon who was also interested in this

ThomasMichon commented 5 years ago

I've been thinking about this idea for a while, and I've thought about the general requirements one would want from a usage perspective.

These days, one opens a VS Code window into the root folder for a rush-enabled repo, and can use the integrated terminal to open the console. Ideally, you can cd around there and run npm run ___ or rush ___ and they should 'just work'.

Suppose you have two repos:

src/git/repo-a/
    packages/project-1/package.json
    packages/project-2/package.json
    rush.json
/src/git/repo-b/
    packages/project-3/package.json
    packages/project-4/package.json
    rush.json

I would expect a flow something like this:

~/> cd /src/git/repo-a

/src/git/repo-a> rush bridge add
Added 'repo-a' to the default bridge.

/src/git/repo-a> cd /src/git/repo-b

/src/git/repo-b> rush bridge add
Added 'repo-b' to the default bridge.

/src/git/repo-b> rush install
This repository is part of the default bridge. To remove this repository, use 'rush bridge remove'.
Initializing pnpm
...yada yada

/src/git/repo-b> rush build
This repository is part of the default bridge. To remove this repository, use 'rush bridge remove'.
Building
[project 1]
[project 2]
[project 3]
[project 4]
Done

/src/git/repo-b> rush bridge remove
Removed 'repo-b' from the default bridge.

/src/git/repo-b> cd /src/git/repo-a

/src/git/repo-a> rush bridge remove
Removed 'repo-a' from the default bridge.

If you let users work from their repo's terminal, then you can make most rush script commands work across the whole bridge. Essentially, they all proxy over to the rush configuration for the bridge, which orchestrates the whole thing.

Now, regarding common-versions.json and other repo-specific configuration, I think the way to get this feature off the ground would be to start with ignoring those features, and see what is required first to make cross-repo linking and command running work.

One thing we could look into doing is making the phantom common/temp project set up by rush actually 'nest' the bridged repositories as their own projects.

~/temp/rush/bridges/default/
    common/temp/
        projects/
            package-1/package.json
            package-2/package.json
            package-3/package.json
            package-4/package.json
        repos/
            repo-a/package.json
            repo-b/package.json
        package.json

When you build the 'root' package.json file, you make it reference only the sub-repo projects, but also and common versions extractable from all the cross-repo packages. But for anything common within only a single repo, put it in the package.json file for that repo, and do the same with entries from common-versions.json. I think this would handle that issue gracefully, at least with respect to not breaking each of the bridged repos. However, ultimately we'd want to 'choose a winner' when it comes to packages that should be common across both repos, so there might need to be some magic involved in produced an aggregate common-versions.json. I was thinking that maybe common-versions.json needs support to declare that specific packages do support wildcards, but only to to resolve discrepancies with another whole repo.

For pnpmfile.js, we can either blindly compose multiple pnpmfile.js values together, or work with the pnpm owner to allow the overrides to be scoped when imported under specific sub-trees.

octogonz commented 5 years ago

Let's start by designing the command-line experience. Sounds like above you're proposing these new commands:

rush bridge add
rush bridge remove

Some feedback:

octogonz commented 5 years ago

Regarding common-versions.json, @ThomasMichon and I chatted and we think there should be an @rush-temp/ tarball representing each repo. This would allow each repo's preferred versions to be maintained separately, while still performing a global install for everything.

calebmshafer commented 3 years ago

@octogonz has this feature request progressed any further than what's detailed in this issue? If not, are there any changes in Rush over the past couple years that you think would make this less feasible?

We are running into the same issue described by @moward in our current use of Rush. We have a main repository (iModel.js) that contains platform/base level packages and then several additional repos containing various packages and apps built on top of iModel.js. This has worked well so far but has been a continuous nightmare when making changes across the repos or down in the platform that need to propagate up (sometimes 3-4 different rush repos).

octogonz commented 3 years ago

With the new "useWorkspaces" setting in rush.json, I feel like this problem has maybe gotten easier. PNPM will do the hard work of linking folders, and settings like common-versions.json are less important.

sammarks commented 3 years ago

I think the correct move would be to update Rush to automatically add those "bridged" packages inside the pnpm-workspace.yaml file, and then run the installation from there (assuming Workspaces allows for something like that), and that should be it?

Moreover, instead of bridging two entire monorepos together (as suggested above), I think it would be simpler on the user's end to just bridge together specific projects (if you want to make changes to both monorepos at the same time like rush add, why are they not already merged into the same monorepo?).

My typical usage is I'm relying on a package from another monorepo I already have cloned locally, and don't want to either rsync or have to publish / update every single time I want to send a change over.

Speaking to the mismatched version detection above, is that something Rush facilitates, or does PNPM facilitate that? If Rush is the one facilitating it, then it should be easy to workaround by just using pnpm-workspace.yaml. You just run the risk of having different versions of common dependencies, but, you would run the risk of that anyway by just depending on the package normally.


My earlier comment, on my very hackish immediate workaround:

So I'm trying something which I feel might be a reasonable (albeit very hacky) solution if you quick-and-dirty want to get packages linked together.

I've noticed that even though it isn't technically allowed to use link: version specifiers for dependencies, PNPM seems to play well with it. So I've basically modified my pnpmfile.js to read the links from a local configuration file, and dynamically update the dependencies of packages in the workspace.

Here's the pnpmfile:

'use strict'

const path = require('path')
const fs = require('fs')

let linkedPackages = {}
const LINKS = path.join(__dirname, '../config/links.json')
if (fs.existsSync(LINKS)) {
  const contents = fs.readFileSync(LINKS).toString()
  linkedPackages = JSON.parse(contents)
}

module.exports = {
  hooks: {
    readPackage,
    afterAllResolved,
  },
}

function afterAllResolved(lockFile, context) {
  for (const importerKey of Object.keys(lockFile.importers)) {
    // Check to make sure it starts with ../../ because that means it's probably
    // inside the current project.
    if (importerKey.startsWith('../../')) {
      for (const root of Object.keys(linkedPackages.roots || {})) {
        for (const linkedPackageName of Object.keys(linkedPackages.roots[root] || {})) {
          const finalPath = path.join('../..', root, linkedPackages.roots[root][linkedPackageName])
          context.log(`(lockfile) ${linkedPackageName} → link:${finalPath}`)
          lockFile.importers[importerKey].dependencies[linkedPackageName] = `link:${finalPath}`
        }
      }
    }
  }

  return lockFile
}

const DEP_KEYS = ['dependencies', 'devDependencies']
function readPackage(packageJson, context) {
  for (const depKey of DEP_KEYS) {
    for (const root of Object.keys(linkedPackages.roots || {})) {
      for (const linkedPackageName of Object.keys(linkedPackages.roots[root] || {})) {
        if (packageJson[depKey][linkedPackageName]) {
          const finalPath = path.join('../..', root, linkedPackages.roots[root][linkedPackageName])
          context.log(`${linkedPackageName} → link:${finalPath}`)
          packageJson[depKey][linkedPackageName] = `link:${finalPath}`
        }
      }
    }
  }
  return packageJson
}

And here's what the configuration links.json (inside common/config might look like:

{
  "roots": {
    "../other-project": {
      "@other-project/package-name": "packages/package-name"
    }
  }
}

That seems to work in that it actually links the dependencies to the proper locations, but I'll note a few things:

Not sure this is the best approach to resolving this issue, but I do like the idea of defining all of the links in a single file and it applies to all packages in the monorepo instead of having to run pnpm link for each and every project.

UPDATE 09 / 24 / 20 - It looks like the initial implementation I had (the piece above minus afterAllResolved), didn't jive too well with Rush. So after some fiddling / looking through the code, I think I have it working reasonably well, although all bets are off as to what happens when you try to publish with active links installed (I would assume, nothing?)

I made the following changes to the Rush codebase to get this to work:

apps/rush-lib/.../PnpmProjectDependencyManifest.ts

    if (!shrinkwrapEntry) {
      // We don't support linked dependencies inside the shrinkwrap file.
      if (throwIfShrinkwrapEntryMissing && !version.startsWith('link:')) {
        throw new InternalError(`Unable to find dependency ${name} with version ${version} in shrinkwrap.`);
      }
      return;
    }

apps/rush-lib/.../WorkspaceInstallManager.ts - inside _createPerProjectManifestAsync(), around line 567.

      // We want to skip linked versions, as we don't have an automatically-computed integrity for them.
      if (version.startsWith('link:')) {
        continue;
      }

apps/rush-lib/.../PnpmShrinkwrapFile.ts - inside _parsePnpmDependencyKey, around line 624

      // We don't support links in shrinkwraps.
      if (!result && !pnpmDependencyKey.startsWith('link:')) {
        throw new Error(
          `Cannot parse PNPM shrinkwrap version specifier: "${pnpmDependencyKey}"` +
            ` for "${dependencyName}"`
        );
      }

My understanding is this part of the codebase parses the PNPM lockfile generated inside the temp directory, and pulls out the integrity fields, storing those in individual shrinkwrap files for each project, and that in turn has something to do with not re-installing all of the dependencies each time you run rush update.

And of course, this doesn't work if you tamper with the versions inside your pnpmfile.js (although the common-versions.json file does the same thing after inspecting its pnpmfile implementation, so I'm not sure if this error comes up or not when using that).

Alternatively, I'm assuming you could get around the code change by disabling the legacyIncrementalBuildDependencyDetection inside experiments.json.


It would be interesting to see if Rush would be interested in supporting that links.json structure; the purpose being you define all of your "external-package" links, and it works with PNPM workspaces to install those links properly.

As for how this would work using potential future workspaces or being integrated into the core linking functionality, I have no idea.

If this is interesting, I could start working on a PR with a new configuration file for defining these and an implementation.