Infer project references from common monorepo patterns / tools

RyanCavanaugh commented 6 years ago

Genesis: see section "Repetitive configuration" in https://github.com/Microsoft/TypeScript/issues/3469#issuecomment-400439520

Search Terms

monorepo infer project references automatically yarn lerna workspace package.json

Suggestion

For common monorepo managers, we should natively understand cross-project references declared in package.json as if they were declared in tsconfig.json

Open questions:

Which formats (lerna, yarn, pnpm, etc) would be supported? Can all of them be consistently detected, or would you need to opt in to a specific "monorepo format" to enable a specific resolution algorithm?
How do we find the tsconfig.json file? This data is actually not present in the current (non-tsconfig) dependency graph. We could assume it to be in the package root; what if it's elsewhere?
Would you need to opt in? Would there be a way to opt out? What should that look like?

Use Cases

Monorepos of all (supportable) flavors

Examples

https://github.com/RyanCavanaugh/learn-a

This repo has a fair bit of duplication where projects need to write down their dependencies in package.json and as references (with different syntax) in tsconfig.json.

Checklist

My suggestion meets these guidelines:

[x] This wouldn't be a breaking change in existing TypeScript / JavaScript code
[x] This wouldn't change the runtime behavior of existing JavaScript code
[x] This could be implemented without emitting different JS based on the types of the expressions
[x] This isn't a runtime feature (e.g. new expression-level syntax)

weswigham commented 6 years ago

Which formats (lerna, yarn, pnpm, etc) would be supported? Can all of them be consistently detected, or would you need to opt in to a specific "monorepo format" to enable a specific resolution algorithm?

All the tools you've listed use normal commonjs module resolution (and consequently use the normal package.json dependencies, optionalDependencies, and to a lesser degree devDependencies fields) - none of them change runtime behavior at at all in that regard. Where they differ is where they symlink things into/from (some install all packages in one dir, then link them into each other; some install everything in a top-level node_modules, since that makes everything available), which is what causes issues for us, usually - our symlink handling needs to be essentially perfect to handle all of these configurations correctly. All that'll solve that is hammering away at it with all the configurations we can find and fixing the bugs we find, IMO.

One of the things I was running into in trying to setup an ui-fabric user suite test (where I symlinked things together in our harness rather than use rush) was that depending on where I symlinked stuff, TS would locate modules in ways I wasn't always prepared for. For example: If I symlink from packages/foo into packages/bar/node_modules, imports from packages/foo/index.d.ts is still going to resolve up from packages/foo, not packages/bar/node_modules/foo (which complicates where you need to place things a bit). These tools all usually handle all these cases right (they have to or the runtime wouldn't work); we just need to continue to be faithful to the commonjs resolver's behavior.

How do we find the tsconfig.json file? This data is actually not present in the current (non-tsconfig) dependency graph. We could assume it to be in the package root; what if it's elsewhere?

I would assume package root, and if we don't find one there, we could read a tsconfig field in the package.json that points at it (cue people asking us to read the configuration straight from the package.json).

Would you need to opt in? Would there be a way to opt out? What should that look like?

The usecase is a (near) zero-config start for common monorepo setups, so opt-out, IMO. Could add something like a --no-pkg flag to tsc -b that stops it from walking package.json files.

bajtos commented 6 years ago

How do we find the tsconfig.json file? This data is actually not present in the current (non-tsconfig) dependency graph. We could assume it to be in the package root; what if it's elsewhere?

In https://github.com/strongloop/loopback-next, we are using monorepo to develop a bunch of modules. Right now, we have one tsconfig.json file per each package. (Because of the way how we are working around missing support for project references, we call this file tsconfig.build.json.)

IMO, it's important to allow each package to have its own tsconfig configuration, because this configuration often involves a list of files to include/exclude from compilation.

Assuming the tsconfig.json file is located in the package root makes perfect sense to me :+1:

Initially, I was reading the proposal as to assume a single tsconfig.json file located in monorepo root, that would not work (at least for us).

Would you need to opt in? Would there be a way to opt out? What should that look like?

If we can find an elegant way how to support most of commonly-used monorepo layout (tools) that are setting up cross-package dependency tree using the information from standard npm/package.jsondependencies, optionalDependencies and devDependencies fields only, then I think TypeScript's build mode should automatically infer project references from package.json too.

I think the tricky question we need to answer first: how to distinguish between dependencies that are considered as monorepo-local and should be configured as TypeScript references; and external dependencies that should be consumed as read-only? The package.json file does not provide any hints on that.

The first solution that comes to my mind:

Find out where is the monorepo rooted, either by looking for lerna.json, yarn config file, etc. or by asking the user to explicitly provide that directory via configuration. I think the explicit config option is a better solution as it does not couple TypeScript with different monorepo solutions and provides a natural place for opting into auto-discovery of project references.
When parsing package.json dependencies, check out which dependencies are resolved as a symlink pointing to a different place inside monorepo - these dependencies should be added as project references. Dependencies symlinked to a different place (outside of monorepo, typically /user/local/lib/node_modules when using npm link manually) (*) or installed as a copy in node_modules should be consumed as read-only.

I find this solution a bit too complex and involved, but don't have any better alternative right now :(

(*) Handling of symlinks outside of monorepo is actually another interesting case to discuss. If we treat all symlinked dependencies as project references, then people using multiple single-repo projects with manual npm link could get benefits from the new build mode too.

For example, loopback depends on strong-remoting, where both modules are maintained by the same team. Sometimes I need to make changes both in loopback and strong-remoting, where the change in loopback depends on the changes made in strong-remoting. To do that, I run npm link path-to-my-strong-remoting-clone in my loopback directory. If we were using TypeScript and the build mode was treating all symlinks as project references, then I could make cross-project renames and rebuild both loopback & strong-remoting in a single build step.

weswigham commented 6 years ago

I think the tricky question we need to answer first: how to distinguish between dependencies that are considered as monorepo-local and should be configured as TypeScript references; and external dependencies that should be consumed as read-only?

I was thinking we could update the -b flag to accept a list of globs (whereas today it takes a single folder or list thereof), so it works like the packages field in lerna or workspace fields in npm/yarn, so you can easily pass tsc -b packages/* apps/* to say "all the packages in these two folders are part of my build". I've found that to be the concise way to express how I've seen these repos laid out when I've been comparing rush, learna, and workspaces - of those, rush is the only one that I've seen be regularly much more explicit than that in its config. IMO, glob support in the input paths are probably worthwhile even if you'd not use package.json reading.

mhegazy commented 6 years ago

how would that work for tsserver?

weswigham commented 6 years ago

@mhegazy AFAIK Nothing in tsserver yet cares about project references (or the dependency graph thereof), just the presence of declaration maps. 😉 The only one that requires it is probably a cross-project compile on save - which is a matter of integrating the entire -b flag into the server - arguments included in some way. Which, for that, we could choose to interpret a top-level tsconfig similar to

{
  "references": [{ path: "packages/*" }]
}

to setup the context. (Which, hopefully, would also allow tsc -b in that directory to work without any arguments). This mimics a lerna.json, a rush,json, or the top-level package.json used for npm/yarn workspaces.

weswigham commented 6 years ago

I mean, jamming it into a tsconfig is ultimately wrong (a .tsbuildrc.json with dedicated build-context wide settings would be more appropriate), since tsc -b's arguments don't correspond to tsc's normal arguments at all, so re-purposing tsc's config file is also wrong (since 99% of it would be meaningless and the remaining 1% would be repurposed overlap), but I've lost that fight already.

weswigham commented 6 years ago

Or alternatively, we can just actually read a lerna.json, a rush.json (maybe), or a top-level package.json with a workspaces field; because why duplicate even that configuration when you don't need to.

Cryrivers commented 6 years ago

I think cross repo reference + yarn workspace worked before TypeScript 2.9

Say you want to import something from your monorepo folder packages/@common/library，the code fix feature in VSCode could correctly suggest the path @common/library in TypeScript 2.8.

However, since 2.9, I believe it goes through symlinks or something, VSCode now suggests something like ../../../../../../node_modules/@common/library, resolving all the way back to the node_modules folder in monorepo root (with yarn workspace enabled).

timfish commented 6 years ago

Also bear in mind that the various tools can treat the workspace globs differently.

I tried pnpm recursive install which goes looking through every subdirectory whereas other monorepo tools I've tried only look at the first level of directories. That meant it also went through a jspm_packages directory we have and tried to install and symlink all those too!

The downside to trying to support specific monorepo types is that these are just the ones we're using this week. Next week we'll probably be using something completely different 😆

friflo commented 6 years ago

I am using a monorepo setup based on lerna. The problem I have in this scenario is, that the root folder contain all dependencies of all packages. This means that every *.ts file is able to import any module, which is in root/node_modules. Before I made this experience I expected that only the package.json dependencies are considered to be resolved. This is what a developer would expect. I think it is sufficient to apply this type of module resolution at compile time via a config setting in tsconfig.json (e.g. "dependencies": "./). The module resolution at runtime (via node) should be unchanged.

dicarlo2 commented 6 years ago

@Cryrivers actually, that bug was introduced in 2.9.2. There's an issue open for it, but I can't seem to find it now. 2.9.1 works as expected.

akosyakov commented 6 years ago

Just to share: for Theia I've added a script that converts yarn workspaces to ts project references: https://github.com/theia-ide/theia/blob/b7471470214533912174fa1c0b07301346026939/scripts/configure-references#L19

It can be executed as prepare script to make sure that they stay in sync: https://github.com/theia-ide/theia/blob/b7471470214533912174fa1c0b07301346026939/package.json#L50

Ciantic commented 6 years ago

I think this setup is a bit backwards, why do we need to define project references in tsconfig.json when they are already in package.json? Is the purpose of this issue to allow "jump to definition/declaration" to work without declaration maps in monorepo/lerna setups?

Imagine following Lerna monorepo setup:

packages/components
packages/adminapp (depends on components)

Each has it's own tsconfig.json, e.g. for adminapp/tsconfig.json:

{
    "extends": "../tsconfig.settings.json",
    "compilerOptions": {
        "outDir": "lib",
        "rootDir": "src"
    }
    "references": [{ "path": "../components" }] // I propose to replace this with more generic packages below
}

Adminapp has the project reference to the components package in the package.json:

{
    // ...
    "dependencies": {
        "@yourcompany/components": "^0.1.1",
    }
}

I propose that instead of having project references, we would have package directories, e.g. In short, instead of this in tsconfig.json:

{
    "references": [{ "path": "../components" }]
}

One could define just the package root directories in tsconfig.json instead:

{
    "packages": ["../"]
}

TS Could look in to that packages directory and search for */package.json files. It would see that there is already package defined in ../components/package.json that provides the @yourcompany/components.

Running a tsc -b -w packages always in the background for declaration maps is a bit annoying, given that setups using webpack/parcel/fuse-box does not even require running tsc in the first place cause they do the building in custom scripts.

It seems a bit odd that monorepo setup can't jump to declaration without a declaration maps. The source code of each project is lying in the packages/PROJECT_NAME/src/ as defined by the tsconfig.json setting rootDir (src/), if there were a way for TS to just look the source code first it shouldn't need to have a declaration maps?

RyanCavanaugh commented 6 years ago

~~Due to the most common monorepo tool effectively blocking our use of it [1], I don't think we'll be able to do any work here without inviting potential legal problems.~~

~~FYI @satyanadella~~

[1] https://github.com/lerna/lerna/pull/1616

Edit: We're back; see below.

yortus commented 6 years ago

@RyanCavanaugh I think it's premature to close this issue. There must be ways of reducing the duplication of dependency info that aren't tightly coupled to any particular monorepo tool.

We only really need two things, neither of which depend on external monorepo tools:

the project dependency graph. This can be computed from the package.json files (that's how the external tools already compute it)
a set of glob patterns to identify candidate projects. This can be specified once at the monorepo root, for example in tsconfig.json.

chriseppstein commented 6 years ago

@RyanCavanaugh You can lock to an older version of lerna for now. The license change isn't retroactive.

RyanCavanaugh commented 6 years ago

I'm not interested in getting sued (or getting my employer sued) because I forgot to put the right characters in a package.json I wrote as part of my job. Could have happened once already.

Regardless, any feature work we do relating to lerna would need to be tested on its latest version. People will log bugs and point us at their repos that point to a latest version of it, possibly recursively in a way that we won't notice. Just running a customer repro for a bug involving lerna will potentially put us at legal risk, because Microsoft will be "using" software it's forbidden from using.

We can re-open this if a maintained ecosystem that is legal for us to test with opens up.

yortus commented 6 years ago

Well the way I see it, if --build mode is useful on its own merits without depending on monorepo tools, then its approach to dependency declarations can be improved without reference to monorepo tools.

The argument here sounds like "we can't consider fixing or improving aspects of --build mode, because some users with issues or suggestions might be using lerna."

RyanCavanaugh commented 6 years ago

This issue specifically is basically "make tsbuild work from the same fields lerna does" even though that's not the title. If lerna and some tool X (let's call it free lerna, or flerna) standardize on behavior so that we can test flerna without getting sued, then that's all well and good. If not, then some other part of the TS ecosystem written by someone not under a bill of attainder will fill that gap.

My current take on the monorepo ecosystem is that there isn't something as popular as lerna yet, and we don't usually want to invest too much in places where we wouldn't be supporting the most-popular variant as a first-class citizen. Tool popularity will almost certainly change in the future and, like all other issues here, we'll reconsider it at that time even if this particular currently has the open/close bit set to zero.

There is plenty in the pipe to fix and improve --build already. It's not going anywhere.

_{_{Also I am trolling a tiny bit because come on}}

yortus commented 6 years ago

Okaaay. Well can we still look at making project dependency declarations less repetitive? Does it need a new issue? I've already had colleages get the tsconfig.json references out of sync with the package.json deps and it leads to annoying problems (which I'm happy describe in a suitable issue).

Also:

People will log bugs and point us at their repos that point to a latest version of [lerna] ... will potentially put us at legal risk

this is gonna happen anyway, it's not specific to this issue. Hopefully that bit was trolling 🤞.

RyanCavanaugh commented 6 years ago

If people feel that the issue needs to be in the Open state to discuss things, then so be it

akosyakov commented 6 years ago

@RyanCavanaugh what about the setup with yarn workspaces without lerna?

Ciantic commented 6 years ago

I don't think this feature need to be linked to lerna at all.

If there were just tsconfig.json setting

{ packages: ["../", "C:/Source/SomeDirectory/"]}

TS could just loop ../*/package.json and C:/Source/SomeDirectory/*/package.json and read those files to see which ones are local packages.

octogonz commented 6 years ago

For the Rush tool, there is an explicit list of packages in the rush.json file at the repository root. (We try to avoid wildcards that recurse directories because they are slow. Also they can pick up false matches: We have many tooling tests whose input is subdirectories with package.json files. They look like NPM projects, but are not meant to be processed by the primary toolchain.)

Is there a generic way to implement this that doesn't require the compiler to be familiar with every tool that manages monorepos?

RyanCavanaugh commented 6 years ago

@RyanCavanaugh what about the setup with yarn workspaces without lerna?

The same people who initiated and merged the license changes in lerna are also committers to yarn, so I would assume we're also "at license risk" with that project. I'd want to have assurances from that project that we won't be banned in the future from using their software - we don't want to get into the state where yarn upgrades aren't testable from our side.

sebmck commented 6 years ago

@RyanCavanaugh

The same people who initiated and merged the license changes in lerna are also committers to yarn, so I would assume we're also "at license risk" with that project.

Jamie has no authority over Yarn and this would never happen.

RyanCavanaugh commented 6 years ago

Thanks @kittens - now I just need to pose the same question to the rest of the OSS world 😅

petermikitsh commented 6 years ago

There is already an MIT fork, but it could too early to know how closely it will track Lerna: https://github.com/LernaOpenSource/LernaOpenSource

davismj commented 6 years ago

Edit: Appreciate your update to your post on the thread. Well done.

@kittens

Jamie asked your permission to make this change to lerna.

I have already spoken to @kittens and @evocateur about this privately, but I do need @kittens to give us permission to make this change.

You gave it to him.

I haven't been involved with Lerna since I first started it. None of the code blames to me so I'm happy to relinquish control and copyright to the existing maintainers.

You claim that

Jamie has no authority over Yarn and this would never happen.

But if you ever got too busy and someone else took over, you would let them make the same change to Yarn, it seems.

I understand where you're coming from and not wanting to get personally involved or take sides, but I would say for the good of the open source community you should have said something more along the lines of:

"This seems like a wholly inappropriate move. However, I haven't been involved with Lerna since I first started it. None of the code blames to me so I have no choice but to relinquish control and copyright to the existing maintainers."

or more boldly

"No. As far as the decision rest with me, I will not allow this change. It is contrary to the spirit of open source."

evocateur commented 6 years ago

I'm in the process of writing an apology to the community (for a number of things) accompanying a revert of the license changes. I have been suffering a low-grade anxiety attack almost all day long. I appreciate your patience.

RyanCavanaugh commented 6 years ago

@evocateur thank you for the extremely well-considered response. Your PR message is something I've bookmarked for future reference because it's a great example of what both leadership and real introspection look like in the context of an earnest misstep, which we are all due to make in time.

waltz commented 6 years ago

@evocateur I'm really glad that you decided to make those original license changes. As a maintainer of an open source library I think that you are well within your rights to make significant restrictions on the use of your software. I think that you made a significant impact by forcing an entire community to think about the political implications of the software that they write. I hope that other maintainers look to your change as inspiration.

bajtos commented 6 years ago

Can we move the discussion back to topic please?

Infer project references from common monorepo patterns / tools

For common monorepo managers, we should natively understand cross-project references declared in package.json as if they were declared in tsconfig.json

panjiesw commented 6 years ago

@bajtos Yes, please. Especially I'm interested in what @Ciantic suggested above. tsc -b could, maybe, detect some pattern of monorepo project to compile them in one bulk. Some detect-able pattern I know

yarn workspace will have "workspaces": [] field in root package.json. Then TS can read this and traverse the dirs, finding tsconfig.json along the way, which almost possibly next to package.json which may contain reference to other sub-package.
lerna + NPM local file reference, see https://github.com/panjiesw/learn-a for example in this. Local sub-packages are specified as "pkg": "file:<pkg-location>" in root's or other sub-package's package.json.
Other not-so-trivial to detect pattern can use a field in some top-level tsconfig.json, like "packages" @Ciantic mentioned above.

donaldpipowitch commented 6 years ago

Happy to see that this issue is still actively worked on. We also use monorepos without Lerna (currently based on yarn, but could be pnpm in the future). We not just use different tsconfig.json's for each package, but also for src/tests directories and so on. Things I'd like to see is cross-package (and cross-directory) refactorings for example.

If you ever need a project as a reference, I could offer https://github.com/Mercateo/ws which show cases our project structure.

chadbr commented 5 years ago

I guess it would be a dream to support .net and typescript in the same monorepo?

We have projects that are essentially cross language "model" packages (i.e. poco's in .net, typescript, etc.)

It would be nice for each folder to switch context... but I know that's some pretty serious work and probably off into huge IDE land.

spion commented 5 years ago

I agree with @weswigham regarding tsc -b. All you really need per package is

a tsconfig.json that doesn't know about references
a package.json that does

I think this would all be easier if tsc --packages glob/**/for/package.json worked, and in package.json a {"tsconfig": "./tsconfig.json"} was supported (thats the default path, if unspecified). Simple and flexible (you can for example maybe not watch some packages that you don't plan on changing by not passing their package.jsons). References are built from package.json dependencies and devDependencies data (if you don't want it included, just don't pass its package.json - but if you do, its probably best that devDependencies are also respected for ordering). It also doesn't rely on any monorepo tool, only on the package.json dependency graph which is fairly universal at this point.

weswigham commented 5 years ago

I actually made a small script (unpublished, ignore the readme) as a proof of concept for discovering project references from package.json files. Problem was every monorepo I looked at inevitably had circular references between the projects which we don't actually support right now. 😄

donaldpipowitch commented 5 years ago

How would you handle multiple tsconfig.json's inside one package in this case? E.g. inside a src/ and tests/ directory? It is currently not possible to have one tsconfig.json and different settings based on file patterns (like ESLint supports) for example.

spion commented 5 years ago

@donaldpipowitch good point, I forgot about that. Jest and mocha can both run TS files with separate configs directly. Do you have any representative examples that can be used to check an idea?

(as an aside, collecting a few heterogeneous examples would actually be pretty great...

@weswigham thats pretty strange, since AFAIK you can't actually publish packages with circular references on npm. I'm guessing they're private projects? Our monorepo tool wsrun detects them and refuses to run if they're present; strangely enough we haven't had complaints so far (possible that people simply give up on it immediately). That script or something like it is probably going to be super useful to us :grin:

donaldpipowitch commented 5 years ago

Do you have any representative examples that can be used to check an idea?

Yes :) https://github.com/Mercateo/ws

This project contains our shared build configuration alongside with some examples which showcase our project structure :)

This package has actually three tsconfig.json's for example: for the source code, for unit tests and for E2E tests.

Our monorepo tool wsrun ...

❤️ wsrun

btakita commented 5 years ago

What is the current status of this issue? I interpret the Open status that the current version of Typescript is unable to infer references.

https://itnext.io/how-to-set-up-a-typescript-monorepo-with-lerna-c6acda7d4559 referenced this issue. Are there any other tutorials or docs detailing an effective workflow with Typescript & Lerna?

If I switch to Typescript, I would also need to be able to load compiled (ts->js) files at runtime for bin/* scripts. Does that mean I need to have a watch script for all libraries? I have hundreds of libraries in active development. Does that mean I need hundreds of watch scripts?

weswigham commented 5 years ago

Correct! However as an aside, you can now get most of the same perf gains as project references may have gotten you in a bigger project via the --incremental flag, hopefully without drastically changing how your project(s) build(s).

itsdouges commented 5 years ago

I actually made a small script (unpublished, ignore the readme) as a proof of concept for discovering project references from package.json files. Problem was every monorepo I looked at inevitably had circular references between the projects which we don't actually support right now. 😄

hi @weswigham - i'm currently looking into moving https://github.com/element-motion/element-motion/pull/144 to use ts project references.

does your script work for tsconfigs that already exist?

weswigham commented 5 years ago

Hm? That old script is written to skip over existing tsconfigs IIRC

SimenB commented 5 years ago

npm has announced that they'll support the same workspace feature that yarn supports: https://blog.npmjs.org/post/186983646370/npm-cli-roadmap-summer-2019. Would be sweet if tsc could understand it as well, seems to be accepted as standard at this point

Bnaya commented 4 years ago

yarn has yarn workspaces info command that outputs the needed info to automatically setup references For example:

  "@thi.ng/rstream-csp": {
    "location": "packages/rstream-csp",
    "workspaceDependencies": [
      "@thi.ng/csp",
      "@thi.ng/rstream"
    ],
    "mismatchedWorkspaceDependencies": []
  },
  "@thi.ng/rstream-dot": {
    "location": "packages/rstream-dot",
    "workspaceDependencies": [
      "@thi.ng/rstream"
    ],
    "mismatchedWorkspaceDependencies": []
  },
  "@thi.ng/rstream-gestures": {
    "location": "packages/rstream-gestures",
    "workspaceDependencies": [
      "@thi.ng/api",
      "@thi.ng/rstream",
      "@thi.ng/transducers"
    ],
    "mismatchedWorkspaceDependencies": []
  },

Bnaya commented 4 years ago

I've created a cli tool based on information from yarn workspaces info (can be made to work with lerna), That injects the needed refs, set composite: true and also includes other automations on tsconfigs on monorepo https://www.npmjs.com/package/typescript-monorepo-toolkit

btakita commented 4 years ago

I solved this using a custom shell script. I have similar monorepo project to @thi.ng/ called @ctx-core/. I tend to use a git submodule to include the packages in active development. @Bnaya, I'm excited about @thi.ng. I'd love to find a way to mix in other large scale monorepo libraries together so we can utilize each others work. I'm sure patterns will emerge. A vision that I see is every developer & organization could cultivate their own monorepo, with a la carte libraries that are under their own control by utilizing forking.

#!/bin/sh
ROOT_DIR=$(dirname $(dirname "$0"))
tsc -b \
  $(ls $ROOT_DIR/packages/ctx-core/packages/*/tsconfig.json | xargs dirname) \
  $(ls $ROOT_DIR/packages/*/tsconfig.json | grep -v ctx-core | xargs dirname) \
  $@

Bnaya commented 4 years ago

@thi.ng is not mine, i'm just making some minor contributions:) yarn 2 supports nested workspaces. not sure exactly how its gonna work, but for sure its gonna be cool

microsoft / TypeScript