Support for multi-repo and reusing libraries from one nx workspace in another via npm workspace or manual symlinks

[jonstelly]

Description

It would be nice to be able to reuse libraries from one nx workspace in another via code reuse (git submodules, symlinks, etc...) where we get the same great inner loop development experience of modifying code in a library without having to rebuild the library and relaunch the nx serve.

There is a blog article from Victor Savkin from a few years ago talking about something pretty close to my scenario but it was before a lot of the nx workspace stuff took shape and it doesn't seem to work now. There is an existing issue talking a bit about NPM workspaces here: https://github.com/nrwl/nx/issues/3843

I have an example repository here which consists of two nx workspaces fw and app, where I want to reuse a library from fw in app. In the real world I have a framework repository and a separate application repository (several different app repositories) and I have a git submodule from each application that refers to my framework repository. So it's very close to a mono-repo but the folder structure is different and I have a separate nx workspace in my framework repository with unit tests, storybook, etc...

Motivation

For those who can't or don't want to move to mono-repo, or maybe have the slightly-different "almost-mono-repo" using git submodules like myself, we would still like to take advantage of all of the other great things about nx workspaces. From my testing it only seems like there's 1 bug/issue in the Ivy compilation that's preventing code reuse of nx libraries across workspaces.

Suggested Implementation

My example repository above uses a manual symlink for simplicity but I would prefer to use NPM workspaces. I did test out NPM workspaces but ended up at the same Ivy symlink issue mentioned in my example repository. It's worth noting there may be more effort to support NPM workspaces, since the tsconfig.json includes use relative paths and would probably point to missing files if they get symlinked under node_modules like they do with NPM workspaces.

I also tried adding the framework libraries to my app's nx workspace by jumping out of my nx workspace (angular.json project referring to directories outside of my app workspace and up to my framework workspace like "root": "../../fw/libs/my/client",) but I got errors about source files outside of rootDirs, etc...

I will find some time in the next couple days to branch my example repository and create the same thing using NPM workspaces, and also using project directory references that jump out of my app's nx workspace.

Alternate Implementations

I'd prefer the NPM workspace option since I wouldn't have to explain, document or script the creation of the symlinks and it would just happen by default when a developer does an npm install, but I'd be happy with manual symlink support (since that would bring the other module under the root of the app's nx-workspace). I suppose referring to projects/libraries outside of my nx workspace root would be OK, but that seems the most problematic.

[fguitton]

If our particular case we have a private monorepo containing all our libs and internal projects. But we also have to maintain separate monorepos with only a subset of projects because collaborations with external companies requires us to give them access to some, but not all libraries (this is because some of these are SDK libs which are co-developed and used we use across projects). Git permissions can be setup very straight-forwardly per git repo, but are not granular within a repo. This forces us to do a lot of copy/paste/manual sync of code-bases which isn't ideal.

The ability to play with gitmodules/symlinking perperly would alleviate these issues and will have added bonus of maintaining coherent branch histories and other goodies.

[github-actions[bot]]

This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs. If we missed this issue please reply to keep it active. Thanks for being a part of the Nx community! 🙏

[fguitton]

Keeping this alive ! 😃

[jaruesink]

We are running into a use-case for this because we are using some open-source repositories that we would like to keep up to date. It's challenging to do that when we have them pulled into our NX file structure, so it would be nice to be able to set it up where NX commands can work in parent directories somehow.

[Nick-Lucas]

We're also looking for a solution to this. We want to retain some separation between applications since the codebase for each app is pretty big, but also want to share a number of libraries between them. Nx having the ability to consume other monorepos seems a sane solution, and bringing them in as submodules also seems sane so you can pin versions easily

[alt1o]

We need this so badly. We currently have two projects both based on nx, one open source and one private, and we want to reduce the complexity of starting up when developing locally

[tzhangchi]

Our team needs this feature very much. We have used NX to manage all projects. NX is great

[jebesampaio]

Same here..without this, local development is close to impossible. Right now we are only able to publish the libs to npm and then use the new published version in the other repo, so testing changes is a painful process.

[todorpavlovic]

Same here, any solution here ?

[vmatyukh]

I'm waiting for this feature.

[moatorres]

Nested Nx Integrated Workspaces with native support for git submodules, please 🙏

[artiphishle]

As far as I understand is using NX = having everything in one huge monorepo, where libs live inside the repo, and apps use (share) different libs. Now this is very not flexible when I have 2 clients (different apps, different repos that share same libs) ?? Probably the feature that's requested here.

I was using meta so far: https://github.com/mateodelnorte/meta

Thought NX might be more powerful (not saying meta isn't, just wanted to try out NX)

Maybe it's easy to achieve, but haven't seen any guide on how to work like that.

Probably that's not NX's job... just git submodules....? (obv. there are 1000 guides on how to add nx, but none how to remove it properly. Hopefully just uninstall package & done, not sure where it puts all the meta data and other things)

[moatorres]

@artiphishle, I have a similar setup and I second that NX helps a lot with it. There's a lint rule where you can enforce boundaries between libraries and applications. I'd suggest to take a look at the official docs. Most specifically: grouping libraries, nx at enterprise and module enforce boundaries articles.

[artiphishle]

@moatorres thank you a lot. Meanwhile I've uninstalled everything to postpone this issue ^^ Maybe I just stick with what I know (Meta repo) for my specific use case. Maybe I will try it again at some point (in a sandbox, where I can try without living in fear). Thanks again, very fast reply!

PS: yeah, in theory that shouldn't be a big issue, I've created the library and everythingn was there (package.json etc) so in theory I just need to add a .git (git init) directory and done (BUT then I wonder what happens if I push the whole thing, would nx automatically know that the library with a .git directory in it shouldn't be pushed, or how to tell) Maybe I was just one command before "great" before uninstalling... Would be nice to get an example for this in the docs (at least I haven't seen one)

[moatorres]

Oh, I misunderstood your requirements, @artiphishle. It may be possible to do what you want (multiple git trees within the same monorepo), but that would be very cumbersome. AFAIK, some of the best features of NX, like the affected command, rely on a single git repository. Maybe you could have a branch for each customer, then configure different remotes and apply rules to those. It would probably involve pre-push hooks and other tweaks. I wonder if we could come up with symbols throughout the monorepo. 🤔

I'm a huge fan of what NX enabled for the ecosystem and would be happy to try and implement something like that to see how it'd work, but I can't provide a timeline. Nevertheless, thank you for sharing your thoughts.

[faheemhameed]

I am very new to mono repo concept. I am just going to try nx in a few mins. I am just creating a customer app and an admin app for the same in angular. I would like to use nx workspace but I would like to keep each application and library in their own git repo. The nx workspace setting I want to keep it in a separate repo and use .gitignore in it to ignore all the apps and libraries within it since each of them has their own git repo. I wanted to keep them in individual repos for different reasons.

I am not sure this is the perfect setup but I would like to try that anyways. I would be happy to get your further suggestions on this.

Thanks!

[istvandesign]

A hybrid multirepo approach would be great to avoid needless rebases between projects when it's actually not needed.