After Nx 11 upgrade, Angular production build sometimes silently fails #4820

Closed rpd10 closed 3 years ago

Current Behavior

We upgraded from Nx/Angular 9 to Nx/Angular 11 (migrated to 10 first, then 11). When running production builds in CI, we are seeing random/flaky failures during the build step, with no information being logged in the console, other than Exit Code 1. Retrying the build step works after 1 or 2 retries. So far this has only happened in GitLab, not on individual developer machines. I'm unsure if this is a Nx issue or Angular issue, but figured I would start here. It seems like it fails during the ES5 build step

Expected Behavior

Successful build, OR console logging indicating failed build

Yes, builds worked consistently in Nx/Angular 9.

Steps to Reproduce

Start with existing Nx 9 workspace using Angular 9
nx migrate to 10
nx migrate to latest
Run nx affected:build --all --prod

Failure Logs

  - ui
———————————————————————————————————————————————
> nx run ui:build:production 
Browserslist: caniuse-lite is outdated. Please run:
npx browserslist@latest --update-db
- Generating browser application bundles...
✔ Browser application bundle generation complete.
- Generating ES5 bundles for differential loading...
———————————————————————————————————————————————
>  NX   ERROR  Running target "build" failed
  Failed projects:

  - ui

  You can isolate the above projects by passing: --only-failed
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
real    2m22.125s
user    3m31.160s
sys 0m7.872s
Cleaning up file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

I tried to turn on verbose logging, but it spits out a ton of messages like harmony import specifier... and then exceeds the 4MB of logging available in gitlab by default.

Environment

This is a local nx report from a developer machine


  Node : 12.18.3
  OS   : darwin x64
  yarn : 1.22.4

  nx : Not Found
  @nrwl/angular : 11.2.12
  @nrwl/cli : 11.2.12
  @nrwl/cypress : 11.2.12
  @nrwl/devkit : 11.2.12
  @nrwl/eslint-plugin-nx : Not Found
  @nrwl/express : Not Found
  @nrwl/jest : 11.2.12
  @nrwl/linter : 11.2.12
  @nrwl/nest : Not Found
  @nrwl/next : Not Found
  @nrwl/node : Not Found
  @nrwl/react : Not Found
  @nrwl/schematics : Not Found
  @nrwl/tao : 11.2.12
  @nrwl/web : Not Found
  @nrwl/workspace : 11.2.12
  typescript : 4.0.7

In CI, we are building with FROM node:12-buster Node: v12.16.2

Can you please check if you have ngcc in your postinstall script?

It should be something like:

ngcc --properties es2015 browser module main --first-only --create-ivy-entry-points

Yes, here is the postinstall script from package.json:

"postinstall": "node ./decorate-angular-cli.js && ngcc --properties es2015 browser module main --first-only --create-ivy-entry-points",

Probably I'm in the same situation about nx test. nx test passes with silent fails since v11.3. It's solved by downgrading version to 11.1.

nx test silently fails at least since 11.2.6, this breaks our CI pipeline, still not fixed. Here is an issue about this (with wrong label, it's a bug, not a question/discussion) https://github.com/nrwl/nx/issues/4711

Sorry for the delay everyone.

I found the issue :tada:

https://github.com/nrwl/nx/commit/f206925abf43c32c83b566a88677555dadb0cff7 fixes the issue and has been released with 11.3.2.

I'm still experiencing this issue with 11.4.0. I found the root cause in our CI environment, but in order to do so I had to opt-out of the NX CLI decoration on top of the NG CLI. I removed the postinstall script, changed to use ng build --prod, and cleared the Gitlab cache to remove node_modules, and then set up a scheduled build to run a bunch of times to try and get it to fail with the plain Angular CLI. When the build fails it logs the following output (truncated):

$ node --version
v12.16.2
$ time yarn build:prod
yarn run v1.22.4
$ ng build --prod
Compiling @angular/core : es2015 as esm2015
...
Compiling @angular/material/slider : es2015 as esm2015
Compiling ng2-charts : es2015 as esm2015
Compiling @angular/google-maps : es2015 as esm2015
Compiling @psu/components/no-permissions : es2015 as esm2015
Compiling @psu/apis/accounts : es2015 as esm2015
- Generating browser application bundles...
✔ Browser application bundle generation complete.
- Generating ES5 bundles for differential loading...
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Killed
error Command failed with exit code 137.
real    2m28.519s
user    4m42.310s
sys 0m11.287s
Cleaning up file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

The Killed...Command failed with exit code 137 means GitLab killed the Docker image being used to run the build, presumably due to inadequate memory allocation for the Docker image.

I also noted that we are wrapping the build command with the "time" command - time yarn build:prod - I am wondering if that is a contributing factor here.

To help debug, I will remove the time command, put back the NX CLI, and run a bunch more builds until I get the OOM error again, and see what is logged.

After removing the time command and turning back on the NX CLI, I still do not see the "Killed" bit of logging:

$ node --version
v12.16.2
$ yarn build:prod
yarn run v1.22.4
$ nx build --prod
> nx run ui:build:production 
Browserslist: caniuse-lite is outdated. Please run:
npx browserslist@latest --update-db
- Generating browser application bundles...
✔ Browser application bundle generation complete.
- Generating ES5 bundles for differential loading...
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
Cleaning up file based variables
00:01
ERROR: Job failed: command terminated with exit code 1

@rpd10 Did you find a solution for this I have migrated the nx version from 13 to 14 and I am getting this same issue

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.

nrwl / nx