Removing implicit dependencies and generating package.json for each project

[atifsyedali]

After using NX, I have a few suggestions for medium-large projects adopting NX. Collectively, they will solve the following issues: #1169, #1777, #602 and probably some others as well.

Affected command and implicit dependencies on workspace.json, nx.json, and package.json

From #1169:

If we introduce a new library every time that we add a new component, our implicit dependencies (angular.json, nx.json, and tsconfig.json) will be updated every time, and therefore all apps are considered affected.

This is a real problem that is affecting medium to large projects. One of the primary motivations to moving to a mono-repo (and hence move to using nx) is to figure out what we need to build and deploy.

I agree with having separate libs (as opposed to doing file-level dependency analysis to figure out what to build), but ultimately any changes to the more freqently changing files -- specifically the angular.json/workspace.json, nx.json, and package.json -- reduces the utility of the "affected" commands.

angular/workspace.json and nx.json

The primary issue here is adding new projects. But sometimes updating existing projects also leads to the same issue.

Proposed solution

1. Have a project.json in every app/lib folder, and this project.json would contain just the parts from angular.json/workspace.json + nx.json that are relevant to that project.

2. An explicit command called nx update-projects (which is also implicity run before any nx command), generate the root angular.json/workspace.json file + nx.json file from all the projects involved (recursive find of all project.json in the apps and libs folders **)

3. Have a top-level property called "generated" in angular.json/workspace.json and nx.json, which would indicate if those files are generated, and subsequently ignore them if they are during affected calculations.

---

** To know the names of all folders that contain apps and libs (even though today in some places in the code it is hardcoded), we can look at the root package.json for an "nx" property (e.g. nx: {appsFolder: "apps", libsFolder: "libs"}).

---

Root package.json

The primary issue here is adding a new external dependency to be used by one of the projects. We now have to build and deploy everything because affected command says so.

Proposed solution

The proposed solution below actually solves two problems: (1) affected command only shows those projects that are affected by the root package.json, (2) a package.json is generated in the build output of all projects, which helps further Dockerization / deployment (See #1777).

1. During the scan of source code to build a dependency graph for the affected command, nx only checks for imports/requires of mono-repo's npmScope. Instead, have it maintain a dependency graph of all imports, including external dependencies.

2. Maintain a subset of dependencies/devDependencies/peer/optional per project in the project's own nxdeps.json file (just like the cache in the output folder). This file would be updated after a call to nx update-projects. The purpose of this nxdeps.json file will be clear in (3b-ii) below.

3. Now, the affected command can be divided into two steps:

   3a) Like before, using the base and head (or affectedFiles), figure out which files were changed in which projects, and then check the dependency graph for all inter-project dependencies (i.e. any that starts with npmScope in the dependency graph).

   3b-i) If base and head were supplied, look at the two versions of package.json and check which dependencies in the dependencies/devDependencies/peer/optional have changed. Compare these with all non-project dependencies in the dependency graph.

   3b-ii) If affectedFiles was supplied instead, then lookup the current values of each non-project import from the dependency graph in the root package.json file, and compare these values (additions, removals, changes in version) with those in nxdeps.json for each project. ***

4. For the build output, generate a package.json for each project using:

   4a-i) Any package.json in the project-root of each project as the base (if it exists -- some users already have it).

   4a-ii) If package.json does not exists in each project, then generate one using all the fields in the pacakge.json in the root of the monorepo ... but do not use the dependencies/devDep/peer/optional. Use the project name as the name field.

   4b) Now using the dependency graph, as well as the externalDependencies configuration option, figure out which dependencies are really external, look up their version value in the root package.json.

   4c) Use (4a) and (4b) above to create a package.json in the build output folder.

   4d) Also, copy over the yarn.lock and package-lock.json in the root to the output folder if they exist.

---

*** If no nxdeps.json was supplied, but affectedFiles was used, give a warning to the user.

---

Update existing nx.json and angular/workspace.json

Finally, remove the following files from nx.json's implicit dependencies: package.json, workspace.json/angular.json, nx.json.

---

Importing publishable libraries into others

The generation of package.json for each project using the above techniques presents an opportunity to hoist individual projects and libraries to publish them to npm/registry. That is, imagine if you could also specify externalLibraries in addition to externalDependencies (or simply are able to detect which libraries are publishable). Then you exclude bundling these libraries during a build, and instead keep their @npmScope/library-name imports intact in the output (i.e. whitelist these imports in nodeExternals).

[FrozenPandaz]

The points and solutions that you outlined are great! Thank you for outlining your thoughts.

We are currently already working on being smarter about how we process changes to package.json. We will be drawing dependencies between projects to node_modules allowing you to update a dependency in package.json and only affect projects which import from that module.

This feature will also come with the infrastructure to be able to parse changes to JSON thus opening up more intelligent processing of other JSON files such as workspace.json, tslint.json, nx.json, .etc!

This is a lot of work and we will be working on improving this during the v10 timeline.

[FrozenPandaz]

Re: points and solutions

Separate project.json files

This would be a much welcomed way of configuring projects. angular.json and workspace.json can be quite large. Generating the workspace.json could work but it would be much better done where the root workspace.json can contain references to other project.json files. We can discuss this with the Angular CLI team to see if they have plans to support this.

The hope with the above changes is that we will be able to intelligently parse the changes one makes to the workspace.json thus not affecting all projects.

Generating package.json

Generating package.json files based off of the dependency graph is an interesting idea. I think the complexity is that even with the full lockfile copied over. Having a different set of dependencies can still result in different versions of installed projects because of scope hoisting. Ultimately it's not much better than writing your own package.json if you need to dockerize the runtime application. There's also a slight distinction between what is needed at runtime of a bundled application vs what the application depends on according the imports in its code. It's a complex problem and for now I think it's better to manually author the package.json files.

These are just my thoughts and I could very well be wrong :).

[studds]

I've written a little script called monorepo-package-tool that copies dependencies from a root package.json to "child" package.json based on imports. It's intended for publishing to npm, rather than dockerizing. (I webpack my code with dependencies for deployment).

It's fairly basic, but it's scratching my itch for the mo.

[atifsyedali]

Thanks @studds -- checking it out. I also have a couple of builders here https://github.com/Apployees/apployees-nx that will traverse the imports and output a package.json in the dist folder for that app/lib. If you have implicit dependencies you can create a partial package.json in each app/lib and it will take precedence over the versions defined in root package.json for specific dependencies of the app/lib.

[jorgekleeen]

I know is not directly related to this, but something I find difficult right now is to know after a while which dependencies in monorepo's package.json are being used in each specific project. What is the recommended way to do make this kind of analysis?

I find this information relevant when you want to remove an old legacy library, and then you want to know which dependencies were specific to that project, so you can get rid of those dependencies.

[FrozenPandaz]

Hi, sorry about this.

This was mislabeled as stale. We are testing ways to mark not reproducible issues as stale so that we can focus on actionable items but our initial experiment was too broad and unintentionally labeled this issue as stale.

[github-actions[bot]]

This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs. If we missed this issue please reply to keep it active. Thanks for being a part of the Nx community! 🙏

[theorlovsky]

i hope this issue is still alive, much wanted 🙂

[jagannathsrs]

Gotta keep this one alive

[dzcpy]

I know is not directly related to this, but something I find difficult right now is to know after a while which dependencies in monorepo's package.json are being used in each specific project. What is the recommended way to do make this kind of analysis?

I find this information relevant when you want to remove an old legacy library, and then you want to know which dependencies were specific to that project, so you can get rid of those dependencies.

Same experince here. When the project becomes huge, it's so hard to find out which modules are not used anymore. To me I'd rather prefer to use a separate package.json in each app / lib. That's probably one of the reasons I have to give up using nx. (Another reason is that currently cli tools have a lot of bugs on windows, I can't even create a project successfully)

[github-actions[bot]]

This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs. If we missed this issue please reply to keep it active. Thanks for being a part of the Nx community! 🙏

[theorlovsky]

any updates on this?

[CormacLynch]

Any update on this would be great

[dzcpy]

Switched to lerna at last. nx brings more trouble than convinience in my projects...

[ianldgs]

Finally, remove the following files from nx.json's implicit dependencies: package.json, workspace.json/angular.json, nx.json.

Did anyone try this? Any luck? Are there any implications with removing the workspace.json and nx.json from the implicit dependencies? Maybe tsconfig.base.json as well... In our case, the package.json is still not a problem.

[thebiltheory]

@FrozenPandaz nudge.

What's the status on this?

I was hoping that nx g remove app-name cleans up the package.json Seems like this solution would solve that issue.

[Frikki]

Dear Nx devs. You really must address this issue along with fixing missing peer dependencies.

[litwin90]

I was wondered that adding only one new lib to the Nx workspace (favorite without actual using it) will triggers affected command to output all the apps and libs as affected and there is no way to avoid it for now (please let me know if I am wrong). So is there any plans to fix this behavior ?

[sdoh55]

@litwin90 I just found out that if you remove package.json from nx.json under implicitDependencies, it will only trigger affected for projects directly using the dependency. This might not be safe for all use cases, so you can also add specific dependencies for specific projects.

[litwin90]

@sdoh55 thank you for the information, will try it out 😊

[johnameyer]

Also curious if there has been any progress here? Using pnpm workspaces so would love for change detection to pick up on what dependencies have actually changed instead of forcing a complete rebuild

[ThomZz]

@litwin90 I just found out that if you remove package.json from nx.json under implicitDependencies, it will only trigger affected for projects directly using the dependency. This might not be safe for all use cases, so you can also add specific dependencies for specific projects.

Working well on my side .. removing package.json -> dependencies from implicitDependencies, to make sure affected apps will be only those importing the changed dependencies.

[ianldgs]

Just a note one this:

I think NX adds those implicit dependencies by default because even though a package might not import a specific dependency, the dependency might be used to build the package. And of course, if you update one dependency that is used to build your lib or app, you want to make sure it's not broken. It would probably be a large effort to track all the dependencies used to build a specific package.

Conclusion: Do this at your own risk! Remember that even if your package is affected by a dep update, it might still get cache.

[github-actions[bot]]

This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs. If we missed this issue please reply to keep it active. Thanks for being a part of the Nx community! 🙏

[AgentEnder]

We've reworked how implicit dependencies work since this issue was filed - they are now part of inputs, and we support package based repositories if folks are against using a single version principle.

[github-actions[bot]]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.