package.json per app

[comunitius]

First of all,. just like to say thank you! I love nrwl nx and how well organized the code looks like when using nx! Also, it's awesome you folks are investing in React!

Prerequisites

Please answer the following questions for yourself before submitting an issue. YOU MAY DELETE THE PREREQUISITES SECTION.

• [X] I am running the latest version
• [X] I checked the documentation and found no answer
• [X] I checked to make sure that this issue has not already been filed
• [X] I'm reporting the issue to the correct repository (not related to Angular, AngularCLI or any dependency)

Expected Behavior

Is there a way we could have a package.json per app? If you are using docker, and would like, let's say, have an angular app running in one container, and the nest api into another container, then both containers will have to run npm install over the all the npm packages even when angular packages are only a subset of all the packages (and viceversa with nest). Additionally, if we do npm install inside the containers, huge packages like cypress will be downloaded, even if they have nothing to do with the api (nest) build. This take its toll especially when running a CI build in travis, circleci, etc (yeah, I know there is cache, but it's not 100% reliable).

Please describe the behavior you are expecting:

Would love to see every app having different package.json. Plus I think it is cleaner. Why having one package,json to rule them all? Seems very tightly coupled.

Current Behavior

What is the current behavior? There is only one package.json for all apps using nx

Failure Information (for bugs)

Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.

Context

Please provide any relevant information about your setup:

• version of Nx used: 8.4.13

[the-ult]

A related question/problem we are facing with using a node graphql-api is:

• how to create small node_modules just for production and just for the node application. Since otherwise node_modules will include the nx, angular, cypress libraries, etc

To solve this, we are using a separate (specific) package.json in the apps/graphql-server.

However, this results in other problems like:

• nx affected:build does not work for the graphql-server (needs to be started from it's own folder)
• npm install order/dependency between main package.json and apps/grahpql-server/package.json
• etc

So the question really is:

• Can we prevent (needing) multiple package.json files when using node applications?

N.B.

• could not find anything in the docs
• We are using separate docker containers as well

[vsavkin]

Thanks for submitting the issue.

You could create a package.json per project, and for instance use something like yarn workspaces to make npm installs relatively fast. We used to do it for one of our internal projects.

We believe in the single-version policy, where, for instance, all applications using NestJS, use the same version of NestJS. Even though it looks a bit crazy at first, we think it is beneficial. Companies like Google do it. So having one node_modules at the root is very similar to what Google does (with a single third_party).

I'll try to find sometime this week to author a guide on single-version policy, where I'll also cover how to add multiple package.json files if you decide that single-version policy doesn't work for you.

[comunitius]

Nice thanks @vsavkin ! In the meanwhile while you create the documentation, do you have an example of a project with different package.jsons?. I just want to decouple my react app from my nest app for now

[the-ult]

Thanks @vsavkin

We love the ONE package.json to rule them all approach and rather have one package.json instead of multiple.

The only reason why we have a separate package.json for our Express (GraphQL-server) app is that we need a small production node_modules for the graphql-server, without all angular dependencies etc.

Would be great if that could be part of the guide.

(each app lives in it's own docker container)

[kylecordes]

Single version policy:

Fantastic, great path for most projects.

Single list of packages:

This is a little harder to deal with as @the-ult suggested. I would love to see something per-app/library that limits which packages (name only, not repeating or varying the version!) the app/library can depend on. This data could also fuel a mechanism to generate a minimal production-only package.json algorithmically when needed.

(I wonder if this might be possible today by omitting package.json from implicitDependencies in nx.dev...)

[wvanderdeijl]

I totally agree with the previous suggestion of generating a minimalistic package.json for production. However, we would also need a package-lock.json to prevent the runtime production app from fetching different versions than the nx workspace.

We mainly need this to get a production app that is as small as possible since it is deployed as a Google Cloud Function

[comunitius]

I'm starting to see the point about one package.json. I've had to deal with multiple lodash versions in the past and it's a pain. Having said that, just like the previous comments, I need a small package.json in production (with the package-lock.json as well) so I can build and deploy multiple small docker images/containers in production.

[aRusenov]

I'm following this discussion and I understand that single-version policy may be an issue for some. But for minimal node server deployments (i.e. not deploying the entire node_modules) why not simply bundle the node app?

[vsavkin]

Quick update: We are working on a course that will cover things like single-version policy. Stay tuned.

[josesaranda]

I think having one package.json per application is useful if you want to publish your libraries or applications separately in the future. Is there anyone who has solved that problem?

[wangzishi]

I'm following this discussion and I understand that single-version policy may be an issue for some. But for minimal node server deployments (i.e. not deploying the entire node_modules) why not simply bundle the node app?

I think it is mainly because there still tons of issues with bundle apps using nx with webpack, nestjs, typeorm, and docker.

https://github.com/nrwl/nx/issues/1518 https://github.com/nrwl/nx/issues/1393 https://github.com/nrwl/nx/issues/1010 https://github.com/nrwl/nx/issues/803 https://github.com/nrwl/nx/issues/633

Each of them is great. But using those significant parts together is terrible.

[mderoche]

Reading through this because I have a use-case with several "sub-applications" that are lazily loaded into a single core Angular application, and they need to be developed by independent teams. I haven't seen that particular wrench thrown in this thread yet.

While the single package.json strategy works great for a single team, I am struggling with how these large companies handle the seemingly incredible cooperation costs of having multiple teams operate with a single package.json.

I am looking for an architecture where we have a core Angular application that is comprised of several sub-applications (which is now facilitated by Nx, and previously by the vanilla Angular CLI.) I wanted each sub-application to have a private package.json to define dependencies specific to that project, while keeping globally shared dependencies (such as Angular itself) in the root-level package.json.

This dramatically cuts the cooperation costs between teams; if Project A needs this-package@1.0.0, and Project B needs this-package@2.0.0, then Project B doesn't need to petition and wait for Project A to upgrade their application to support this-package@2.0.0.

Of course, we still have the issues of "Okay, everyone upgrade to Angular N+1 on mm/dd/yyyy!", but I think having private package.jsons can remove a vast majority of related versioning issues.

This all works fine for JIT compilation. Issues arise when you try and build it with AoT.

Is there a recommendation on how to handle this type of situation, with multiple teams working on individual sub-projects under the same global project?

[demisx]

I like the idea of having a single master package.json file for enforcing versions across the entire repo, but I’m still struggling to understand how one creates a docker image for each app with the package.json being a subset from the master file? How do you derive an appropriate version of package.json per each app? 🤷🏼‍♂️ Obviously, I don’t need angular packages in my NestJs container and vice versa.

[kraiz]

I use generate-package-json-webpack-plugin to solve this problem. But I would like to generate a yarn/npm lockfile instead to tighten the dependencies but didnt find a solution for this yet.

[demisx]

@kraiz Thank you for the tip. Do you know if this plugin can be used for Angular apps as well to weed out the backend (NestJs) dependencies? In other words, is it smart enough to parse Angular source code and build a list npm packages that app needs?

[kraiz]

You don't need it for angular because it's webpack configuration is already making sure you only get the stuff you need into your bundle files without the need for an external runtime dependency folder (node_modules).

[demisx]

@kraiz Got it. That makes sense.

[bennymeg]

I use generate-package-json-webpack-plugin to solve this problem. But I would like to generate a yarn/npm lockfile instead to tighten the dependencies but didnt find a solution for this yet.

@kraiz , Is this what you are looking for?

[kraiz]

Ahh, didn't know that the lockfile can be a superset. Thx, will try that!

[MaximeBernard]

How does Google manage to update all (i can imagine they have hundred/thousands) apps at once?

From my experience, you always get:

• one app that requires too much effort/cost/time to be updated (i.e. that won't happen)
• one app that needs to do something before that (i.e. delaying the update for months/years)
• one app that requires to stay at some package version (angular users are often locked in older versions of TS, or some webpack users often rely on some webpack plugin which has not been migrated yet (or never will))

I'm pretty sure I didn't cover all the scenarios but they all lead to late migration / no migration at all.

Even though I agree it's important to encourage every app/team to stay up to date with the latest releases and use the same version, it becomes more and more complex over time until it's not even possible anymore.

One strategy here for nx could be to offer both options:

• 1 package.json (current way)
• multiple package.json (yarn workspaces way)

That being said, it's probably a lot of work to maintain both possibilities and much more work for many features.

On the other hand, keeping the yarn workspaces way and doing some checks when running nx could offer both worlds features with a limited maintenance overhead.

Whatever you decide, keep up the good work 👍

[mderoche]

@MaximeBernard Are you using Angular with multiple package.jsons? If so, how do you manage to get AoT compilation to work with multiple package.jsons?

[mikkhait]

Just following a thread and wonder if any progress has been made on this front. We are architecting a set of microservices (NestJS, React) and would love to have a package.json for each one of them as well as have a Dockerfile for each app.

Would love to hear how others have solved that?

[MaximeBernard]

Currently, I just use yarn workspaces.

I'm missing the whole part about changes detection, deps graph and retesting/rebuilding only what's affected but I can afford to build the whole monorepo.

Also, using "paths" (e.g. https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#onpushpull_requestpaths) I should be able to smartly mitigate the issue.

[gavindoughtie]

How does Google manage to update all (i can imagine they have hundred/thousands) apps at once?

From my experience, you always get:

• one app that requires too much effort/cost/time to be updated (i.e. that won't happen)
• one app that needs to do something before that (i.e. delaying the update for months/years)
• one app that requires to stay at some package version (angular users are often locked in older versions of TS, or some webpack users often rely on some webpack plugin which has not been migrated yet (or never will))

I'm pretty sure I didn't cover all the scenarios but they all lead to late migration / no migration at all.

Even though I agree it's important to encourage every app/team to stay up to date with the latest releases and use the same version, it becomes more and more complex over time until it's not even possible anymore.

One strategy here for nx could be to offer both options:

• 1 package.json (current way)
• multiple package.json (yarn workspaces way)

It seems to me that you could have an automated package-pruning step using something like https://www.npmjs.com/package/depcheck, and generate per-deployment package.json files at will. My ideal flow would be the current nx approach while developing, but then when I create a docker image to deploy some app, it would traverse the dependencies and produce a minimal package.json for that app image.

[angelnikolov]

Has anyone found any clean solution on this? It's the only thing stopping from fully updating the monorepo approach :( I realize the potential and benefits the single repository gives us, however installing dependencies for 10 separate projects on every docker build is getting slower and slower..

[atifsyedali]

We replaced the default @nrwl/node builder with our own webpack-based node builder, which includes generating a package.json file. It's config-compatible with the @nrwl/node builder. To use:

yarn add @apployees-nx/node

Then find-and-replace @nrwl/node in workspace.json with @apployees-nx/node.

Run a build nx build myproject. Check your dist folder for package.json.

Also, if you have a package.json in the app you are building, it will be merged.

See https://github.com/Apployees/apployees-nx/#enhanced-node-builder----apployees-nxnode

[vajahath]

I was also quite concerned (scared) about the single version policy, initially. But after some read, experiments and thought, I feel like we should follow this single version approach. As the maintainers said, this is like how Google operates and there are valid reasons for it.

But this doesn't mean we've to install every single dependency specified under this repo. Rather, we should use a single version of a package/code at a time across the repo for easy maintenance in long run (how to do it is another problem). This reduces complexity, increases code sharing and tackles many other problems like diamond dependency problem(see below).

This article is a good read. In it they say:

A single repository provides unified versioning and a single source of truth. There is no confusion about which repository hosts the authoritative version of a file. If one team wants to depend on another team's code, it can depend on it directly. The Google codebase includes a wealth of useful libraries, and the monolithic repository leads to extensive code sharing and reuse.

The Google build system5 makes it easy to include code across directories, simplifying dependency management. Changes to the dependencies of a project trigger a rebuild of the dependent code. Since all code is versioned in the same repository, there is only ever one version of the truth, and no concern about independent versioning of dependencies.

Most notably, the model allows Google to avoid the "diamond dependency" problem (see Figure 8) that occurs when A depends on B and C, both B and C depend on D, but B requires version D.1 and C requires version D.2. In most cases it is now impossible to build A. For the base library D, it can become very difficult to release a new version without causing breakage, since all its callers must be updated at the same time. Updating is difficult when the library callers are hosted in different repositories.

This feel like the article not only stresses the point of single version policy, but also adding it as a point in the definition of good monorepos.

When thinking about dependency problem in monorepo, we've to consider code sharing in the first place. Single version policy encourages it.

At the same time, we need to find a way to reduce node_modules size per project while deploying as well as developing. Thoughts?

[MaximeBernard]

Don't get me wrong, I'm deeply in favor of a single repository. This does provide a unified versioning between internal dependencies. But that does not mean you need to have a unified version for your external dependencies.

You should strive for it but the bigger your repository gets, the harder it becomes until it's not even thinkable. And I'm pretty sure Google has already hit that point.

Yarn workspaces and Lerna do just that and do it well.

I'm just suggesting nx to take that approach as well so it meets everyone's needs.

[manju-reddys]

@atifsyedali You done great job, your module (schematic) does extract what the runtime needs after prod build of an app.

[tmtron]

I am a bit confused about this. It seems that different things are mixed up in this discussion.

I'll try to sort my thoughts - please comment when I got something wrong:

Monorepo: everyone here agrees a monorepo is a good thing and this is why we use nx

Dependencies: we have 2 kinds of dependencies:

1. internal dependencies: apps and libraries in the monorepo that reference each other

   • with nx we can directly reference the libs. Then the nx builder will take care to (re-)build what is required
     • i.e. we don't need to manually find the referenced libs, run the build process and install the lib in a local (or external) npm repository
   • nx also allows us to enforce dependency constraints between our libs
   • we always have only up-to-date version of internal libs (i.e. no version numbers needed)

2. external dependencies: downloaded from the npm registry (when we execute npm install)

   • they are currently defined in a single package.json file at the project root
   • external dependencies can have different versions: see version management below

Version management I think we have 2 groups of users here. some who want to;

1. enforce that all code in the monorepo MUST use the same version of each dependency (what nx offers now)
2. have defaults versions, but still be able to override those for some libs

External dependency constraints

The features that I am missing in the current nx implementation:

• we have no way to specify constraints for external dependencies: I'd like to constrain the use of external libraries
  i.e. currently every lib can easily import any external dependency and there's no way to forbid this.
  So a client-lib (e.g. for angular) can accidentally import a huge lib which is only meant to be used on the server-app (e.g. node-js).
• another disadvantage of the single package.json file is that the IDEs will always suggest all code in every lib: e.g. for auto-import, code-completion, etc.
  • if we had multiple package.json files (e.g. the lerna way), then the IDE will find the nearest package.json file and only suggest the "allowed" code

I must admit that I don't have too much experience with nx, npm, yarn, lerna, etc..
But ideally, we'd have one configuration to specify all external dependencies and their versions and per lib a config to specify the allowed external dependencies for this lib (version should be optional: when not set, use the default, otherwise override it).
Then maybe the package.json files can be generated from this config - or maybe we can have an EsLint rule to enforce it?

BTW: Just a hint to the some people who mentioned that they want to have multiple package.json files to reduce the production node_modules folder: you may want to check out the ncc compiler: it can build a single js file which includes your code and only the required parts of the node_modules (and you can optionally minify the output to make it even smaller)

[admir86]

I'll try to find sometime this week to author a guide on single-version policy, where I'll also cover how to add multiple package.json files if you decide that single-version policy doesn't work for you.

hey @vsavkin, any updates on this? any example how to add multiple package.json?

[tomepejo]

@admir86, you can try using yarn workspaces. That worked for me.

You can read a bit here and follow this simple example to do the setup. Then in your Docker file, you should have something like this:

ARG APP_NAME

ENV SERVICE_DIR /usr/src/service/apps/${APP_NAME}
ENV ROOT_DIR /usr/src/service

# Copy root package json
COPY ./package.json ./yarn.lock ${ROOT_DIR}/

# Copy package json of specific service
COPY ./apps/${APP_NAME}/package.json ${SERVICE_DIR}/

WORKDIR ${SERVICE_DIR}

RUN yarn install --focus --production

The last command will install the needed dependencies only for the target application, thus avoiding huge node_modules folder inside your Docker image.

I hope this tip will be helpful for you.

[admir86]

hey @tomepejo thx. the part "Dealing With Version Conflicts" is exactly what I'm looking for. Have you already tried it within a nx workspace? I think I would missing some good features from nx if I had to migrate to a "pure" yarn workspace.

[tomepejo]

Have you already tried it within a nx workspace?

Yes, it works all together. Just give it a try, actually it's quite easy to configure.

[guibulator]

I was actually looking for a solution to a similar issue. I am currently building a single-version policy mono-repo with angular and nestjs with some azure functions created as lib that I wanted to be self-contained so that they can be deployed as the smallest package possible. I don't know why it was not yet mentioned in this thread but this solution is quite easy to use. It leverages the @nrwl/node:builder using a custom web pack config and the GeneratePackageJsonPlugin.

You can follow this example posted by Thomas-jakemeyn

[tomepejo]

@guibulator, that was my first implementation as well, however there are some downsides that lead me to try out the yarn workspaces solution.

• With the package json generate plugin, you are not aware which packages are used by which of your apps, until you trigger the build and check what was generated in the dist folder.
• It works in 90% of the cases, but if some implicit dependency wasn't automatically picked-up, you need to manually add it in the webpack configuration (mysql for example).

Having one root package.json with the common dependencies and separate one in each of your apps (with yarn workspaces) is much more obvious and clean solution. You can be always sure, which apps will be affected if you, add, remove, or update a dependency.

However, the best approach will probably be to try both solutions yourself and decide whatever works best in your case.

[METACEO]

Ahh, I was not aware of this issue/thread last week when I asked my question during the Nx Office Hours: https://youtu.be/6nTroiS3tZA?t=3199 I'll have to iterate and throw some ideas here as this is looking to be a growing issue.

[METACEO]

How in any way could this issue relate with #2483? If we're defining what specific deps affect our apps or libs then could this same definition be used to build custom package.json files?

For example, if I have the below package.json file, would an nx.json file like below help build separate package.json files or identifying runtime dependencies?

While this may require some manual defining by the developers, it would only be whenever a new app, lib, or dependency is introduced. Similar I supposed to Bazel's build files - though if the code could be statically analyzed then this could be automated?

{
  "name": "app-monorepo",
  "dependencies": {
    "@angular/common": "9.0.0",
    "@nestjs/common": "7.0.0",
    "rxjs": "6.5.0",
    "uuid": "7.0.0"
  }
}

{
  "npmScope": "app-monorepo",
  "projects": {
    "app-frontend": {
      "dependencies": {
        "package.json": {
          "dependencies": [
            "@angular/common",
            "rxjs"
          ]
        }
      }
    },
    "app-backend": {
      "dependencies": {
        "package.json": {
          "dependencies": [
            "@nestjs/common",
            "rxjs",
            "uuid"
          ]
        }
      }
    }
  }
}

[sketchnotes]

@METACEO, imho, if it's supposed to be a different file nx.json maybe it might be less redundant and a bit easier to scale up? The Gradle is not totally appropriate way to give the examples because of the different ecosystem but its way to manage the dependencies is rather elegant and scalable. Maybe there's sth to borrow...

[tmtron]

Maybe it should also support js/ts files, so that we can run arbitrary code to build the config/package.json?
This is used e.g. by EsLint and works well.

When you only have a declarative json/yml, ..whatever file, you are very limited - we can see the same problems in angular.json which just grows out of bounds, has lots of duplication and becomes unreadable and unmanageable for larger projects.

[METACEO]

Hey @sketchnotes, regarding the nx.json file I proposed above, I'm not sold on it or tied to it and I remain open to anything that helps solve the original issue (I again like the "unit" approach that Bazel takes!) If I need to throw out questionable ideas to keep the conversation going, then I just may! ;)

I am however not fully understanding your question, could you elaborate?

[westmark]

For what it's worth, I leveraged the fact that all the information you need is located in dist/nxdeps.json and put together this quick-n-dirty script

https://gist.github.com/westmark/9bd341c0bffdf0136969734da7b176e4

For an app located under dist/my-app

> tsc
> node nx-generate-deps.js --app my-app

Anyone using it would (probably) have to change the paths (root mainly). I'm quite new to NX and haven't explored much of what the CLI is capable of doing, but this works beautifully for me in any case.

One obvious downside is the lack of lock files. I have no idea, and haven't tried, if one can just copy whatever is in the project root and use it, even if it will contain extraneous package definitions.

[billyjov]

@tomepejo can you actually pls share an example how to use it in the nx context ?

[ivawzh]

We are hesitated to migrate to NX from generic Yarn workspaces set up due to this issue. Hope in the future, there will be a solution for NX to isolate app dependencies during per-app build. I don't think bundle will help us, because we are using NextJs with dynamic imports and many static assets.

[METACEO]

One obvious downside is the lack of lock files. I have no idea, and haven't tried, if one can just copy whatever is in the project root and use it, even if it will contain extraneous package definitions.

@westmark I agree, I'd prefer not to get into the business of parsing locks files myself!.. but what you're pointing out shows there's a lot of data in the nxdeps.json file to overlap with our package files. It'd be great for more Nrwl input here as we can look at and interpret the nxdeps.json file ourselves, but we may not have the full vision. Ultimately I think there's a lot of data and a lot of opportunities here!

[optimasquad]

@METACEO Do We have any example to follow for per app package.json file in Nx workspace?

[joshmeads]

Building upon @westmark's great work: https://gist.github.com/joshmeads/8740641292d6d336ea3b791a18282c7b

• Fixes a change in recent NX versions where dep.target is prefixed with npm:
• Changes the CLI usage slightly.
  • --appFolder path/within/apps
  • --appVersion 1.0.2 - Optional, sets the package.json version
  • --rootPath . - Optional, sets the root path for the project
  • --docker true - Optional, turns on a flag to change some settings for docker build stages

Usage CLI: npx ts-node ./tools/nx-generate-deps.ts --project ./tools/tsconfig.tools.json --appFolder api/user --appVersion 1.0.2

Usage Docker:

# Root monorepo context

WORKDIR /app
COPY ./dist/apps/$APP_FOLDER ./dist
COPY ./dist/nxdeps.json ./tools/docker/tsconfig.json ./tools/nx-generate-deps.ts ./
COPY package.json ./package-base.json
RUN npx ts-node ./nx-generate-deps.ts --project ./tsconfig.json --appFolder $APP_FOLDER --appVersion $APP_VERSION --rootPath /app --docker true

# Output with package.json will be in /app/dist for the next build stage to copy over.

All of this was built just for my own use case, but hopefully, it helps someone else too. Similar limitations around the lock files though.

[westmark]

Awesome @joshmeads! Maybe this thing deserves its own repo and npm cli package to make it easier to find.

[RogierdeRuijter]

I would like to share the (sub-optimal) solution I found to the problem of keeping the docker images small.

Angular

For Angular I used a multi-stage build to keep the image small.

NestJS

For Nest, it was more difficult because you need to keep the node_modules folder in the image, but you would like to remove all of the Angular dependencies.

To do this I used 2 techniques:

• depcheck with npm uninstall
• npm prune --production

With depcheck you can find all the dependencies that are not used by your code. When you have the list of unused dependencies you can remove them using npm uninstall. I used the following script for this

remove-unused-deps.sh

#!/bin/bash
depcheck > data.txt

cat data.txt 

cat data.txt | (while read F
do
    if [[ ${F:0:1} == "*" ]]; then 
        OUTPUT="$(printf "$F" | cut -c 3-)"
        foo="${foo} ${OUTPUT}"
        # echo "pruned ${OUTPUT} from node modules"
    fi
done && npm uninstall $foo);

After running this script in the docker image I run npm prune --production. To remove all the dev dependencies from the project.

This makes the images smaller, but not as small as if you would have separate package.json files.

Hope this helps someone :)

Cheers

[ivawzh]

@RogierdeRuijter very cool approach, thanks for sharing. And I have a few questions -

Comparing your remove-unused approach to @joshmeads 's pick-used approach. I am assuming the remove-unused approach can support lock file properly, am I right?

This makes the images smaller, but not as small as if you would have separate package.json files.

What is the reason of that ☝️ ?

Does depcheck support peerDependencies removal/keep properly?

---

By the way, I think type: docs is a wrong label for this issue, we want an officially supported feature to keep app size scalable. As @METACEO suggests

I'd prefer not to get into the business of parsing locks files myself!.. but what you're pointing out shows there's a lot of data in the nxdeps.json file to overlap with our package files. It'd be great for more Nrwl input here as we can look at and interpret the nxdeps.json file ourselves, but we may not have the full vision.