Closed robertIsaac closed 1 month ago
clydin
commented
6 months ago This appears to be a package manager related issue. The Angular CLI does not control the layout or content of installed packages. Does the issue occur if another package manager is used?
robertIsaac
commented
6 months ago This appears to be a package manager related issue. The Angular CLI does not control the layout or content of installed packages. Does the issue occur if another package manager is used?
We never tried But I can raise this with the team Which package manager do you recommend going to?
I want to add we upgrade from yarn 1.x to 4.0.2, and the issue still exists
kristofdho
commented
4 months ago @clydin Could you elaborate on how this is a package manager issue? The angular-cli project seems to be using yarn as well, so seems a bit odd to me to blame this issue on yarn while the project is on it as well.
robertIsaac
commented
4 months ago hi @clydin I confirm it's not a yarn issue
- job: Build
dependsOn:
- Cypress_Tests
- Format_Checking
- Unit_Tests
pool:
vmImage: ubuntu-latest
steps:
- task: NodeTool@0
inputs:
versionSpec: '20.x'
displayName: 'Install Node.js'
- script: |
corepack enable
yarn install --immutable
displayName: 'install dependencies'
- script: |
yarn nx run web-app:build:prod
retryCountOnTaskFailure: 3 <<<<---------
displayName: 'nx build'
- task: PublishBuildArtifacts@1
inputs:
pathToPublish: 'dist/apps/web-app/browser'
artifactName: Fusion.Dashboardthis is my build script, it fails now then on retry 1 or 2 times it passes
if it was yarn it wouldn't have been possible for it to pass in on a retry since it's the same node_modules
alan-agius4
commented
4 months ago Unfortunately without a reproduction it's hard to debug and understand what is the problem here.
robertIsaac
commented
4 months ago @alan-agius4
how can I reproduce it if it happens 5% of the times? if you want to get access to our repo I can talk with my manager about the procedure, you (or anyone from the Angular side) will most probably need to sign NDA
florian-wagner-dev
commented
4 months ago The same problem sometimes occurs in my GitLab pipeline, approximately 1% of the time.
alessandro-verdura-maersk
commented
3 months ago We have the same problem. Same as https://github.com/angular/angular-cli/issues/25018 and https://github.com/angular/angular-cli/issues/26389.
We are also getting this issue here:
In addition to the errors similar to posted in https://github.com/angular/angular-cli/issues/25018 ("Undefined variable", "Undefined mixin", "Module loop"), there are also warnings like this
[WARNING] Comments in CSS use "/* ... */" instead of "//" [js-comment-in-css]
We've tried both to point @use directly at the specific partials instead of @angular/material (https://github.com/angular/angular-cli/issues/25018#issuecomment-1593724769), and to tweak the stylePreprocessorOptions setting (https://github.com/angular/angular-cli/issues/25018#issuecomment-1619151058), but neither work.
This never happens in local, only in CI.
It seems that the problem might be related to the typography partials?
I saw some suggesting to make a local copy of whatever typography function you need in your styles, but I feel it's highly impractical, both for the daisy-chain effect of functions you would have to copy, and also in terms of maintainability.
anter-x
commented
3 months ago We have the same problem.
./src/app/common/app-footer/app-footer.component.scss?ngResource - Error: Module build failed (from ./node_modules/sass-loader/dist/cjs.js): Can't find stylesheet to import. ╷ 26 │ @use '@material/feature-targeting/feature-targeting'; │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ╵ node_modules/@material/fab/_extended-fab-theme.scss 26:1 @use node_modules/@angular/material/button/_fab-theme.scss 3:1 @use node_modules/@angular/material/core/theming/_color-api-backwards-compatibility.scss 3:1 @forward node_modules/@angular/material/_index.scss 12:1 @use src/app/common/app-footer/app-footer.component.scss 2:1 root stylesheet
each time it fails on different stylesheet.
Angular version is 18.0.2
jp-sft
commented
2 months ago I encountered same error while building my Angular application.
35.36 - Building...
35.36
84.96 Application bundle generation failed. [49.600 seconds]
84.96
84.98 ✘ [ERROR] Can't find stylesheet to import.
84.98 ╷
84.98 26 │ @use '@material/feature-targeting/feature-targeting';
84.98 │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
84.98 ╵
84.98 node_modules/@material/line-ripple/_line-ripple-theme.scss 26:1 @use
84.98 node_modules/@material/line-ripple/_line-ripple.scss 29:1 @forward
84.98 node_modules/@material/line-ripple/_mixins.scss 24:1 @use
84.98 node_modules/@material/textfield/_text-field.scss 36:1 @forward
84.98 node_modules/@material/textfield/_mixins.scss 24:1 @forward
84.98 node_modules/@material/textfield/_index.scss 2:1 @use
84.98 node_modules/@angular/material/core/tokens/m2/mat/_form-field.scss 3:1 @use
84.98 node_modules/@angular/material/core/tokens/m2/_index.scss 20:1 @use
84.98 node_modules/@angular/material/core/tokens/_m3-tokens.scss 6:1 @use
84.98 node_modules/@angular/material/core/theming/_definition.scss 6:1 @forward
84.98 node_modules/@angular/material/_index.scss 7:1 @use
84.98 projects/<some_path>.scss 1:1 root stylesheet [plugin angular-sass]
84.98
84.98 Any fix for that?
MelanieW97
commented
2 months ago Same here. I was hoping the issue would be fixed with Angular 17 but it's still there. Not on every single build, though. It's quite flaky as some have already said. I'm quite disappointed that this issue does not seem to get any attention at all ...
alan-agius4
commented
2 months ago Unfortunately, there's not much we can investigate without a reproducible example, making it difficult to determine the cause of the issue. It's also worth noting that in our end-to-end tests, which cover various scenarios and utilize both NPM and Yarn as package managers, we have not encountered this error.
Does this also happen when not using NX but using the @angular/cli directly?
tanner-wisniewski
commented
2 months ago Does this also happen when not using NX but using the @angular/cli directly?
My team has been seeing this same issue, using just the angular/cli directly. Like everyone else has mentioned, it only happens in CI, never locally.
One observation is could this be based on the CI platform? Someone called out Gitlab and the screenshot in the original issue looks like it is from Azure DevOps.
kristofdho
commented
2 months ago @tanner-wisniewski we're seeing it on GitHub Actions. So seems to happen across the board.
MelanieW97
commented
2 months ago I use the @angular/cli directly, too. If I simply build the application myself by "ng build --configuration production", everything's fine. The error occured not even once doing so. However, the error occurs in CI and if I try to build a docker image locally. CI breaks way more often than the local docker image build, though. No idea if that information helps locating the problem.
robertIsaac
commented
2 months ago I think it's related to the technical parameters of the builder, in azure it's 2 core + 7 GB ram and 14 GB disk In developer machines it's much higher than that I will try to setup a VM with low core and ram to see if it actually happen there
robertIsaac
commented
2 months ago Unfortunately, there's not much we can investigate without a reproducible example, making it difficult to determine the cause of the issue. It's also worth noting that in our end-to-end tests, which cover various scenarios and utilize both NPM and Yarn as package managers, we have not encountered this error.
Does this also happen when not using NX but using the @angular/cli directly?
@alan-agius4 Tell me how I can help you to investigate?
bmarcotte
commented
2 months ago Just in case this helps anyone else, we have been encountering the same issue in the same manner: only in our CI builds in AWS and never when building locally (mostly macOS ARM dev envs outside of containers). Yesterday, we had a bit of a breakthrough, and we were finally able to reproduce the failure in a local Docker container, albeit only intermittently.
We just completed a batch run of local builds to compare 4 different Docker Node images: node:18.20.3-alpine3.19, node:20.14.0-alpine3.19, node:18.20.3-slim, node:20.14.0-slim. With 20 runs of each of the 4 images, the results were 5 total failures: 4 in Node 20 Alpine, 1 in Node 18 Alpine, and 0 in either Slim image. Our codebase, which contains ~500 components, and a roughly similar # of scss files, was constant across all builds. Only difference between the builds was the base docker image version.
We have changed our CI builds from using alpine to slim, and so far, the results are looking much better. We don't have any hypothesis yet as to why an alpine image would lead to this issue, or why it would only do so intermittently, but at this point, we have enough data to want to avoid them in our environment.
robertIsaac
commented
2 months ago Hi @bmarcotte can you share this setup? I'd like to try this with an empty angular project to see if the size of the project matter at all or not
bmarcotte
commented
2 months ago Hi @bmarcotte can you share this setup? I'd like to try this with an empty angular project to see if the size of the project matter at all or not
@robertIsaac I would only be able to share the docker files & and small bash script that invokes them. Everything else is proprietary code that couldn't be shared as is.
Which I think really gets at the impasse we're at with investigating this. If this problem is more likely to occur in large codebases, but most of our codebases are proprietary, then most of the people reporting the issue are not going to be able to submit a reproducible example, like the Angular team keeps asking for.
So, if anyone happens to know of a large, open source application written in Angular, maybe we could use that as a reference platform for testing issues like this?
robertIsaac
commented
2 months ago @bmarcotte
docker files & and small bash script
that's what I meant
if anyone happens to know of a large, open source application written in Angular
what about Angular material?
anter-x
commented
2 months ago Just in case this helps anyone else, we have been encountering the same issue in the same manner: only in our CI builds in AWS and never when building locally (mostly macOS ARM dev envs outside of containers). Yesterday, we had a bit of a breakthrough, and we were finally able to reproduce the failure in a local Docker container, albeit only intermittently.
We just completed a batch run of local builds to compare 4 different Docker Node images: node:18.20.3-alpine3.19, node:20.14.0-alpine3.19, node:18.20.3-slim, node:20.14.0-slim. With 20 runs of each of the 4 images, the results were 5 total failures: 4 in Node 20 Alpine, 1 in Node 18 Alpine, and 0 in either Slim image. Our codebase, which contains ~500 components, and a roughly similar # of scss files, was constant across all builds. Only difference between the builds was the base docker image version.
We have changed our CI builds from using alpine to slim, and so far, the results are looking much better. We don't have any hypothesis yet as to why an alpine image would lead to this issue, or why it would only do so intermittently, but at this point, we have enough data to want to avoid them in our environment.
We use github actions and tried to run build as docker run --rm --volume "$PWD":/home/node/app --workdir /home/node/app node:20-slim npm run build -- --configuration production --localize and, unfortunately the same result
./src/app/module/homepage/homepage.component.scss?ngResource - Error: Module build failed (from ./node_modules/sass-loader/dist/cjs.js): Can't find stylesheet to import. ╷ 32 │ @use '@material/floating-label/mixins' as floating-label-mixins; │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ╵ node_modules/@material/notched-outline/_notched-outline-theme.scss 32:1 @forward node_modules/@material/notched-outline/_variables.scss 24:1 @use node_modules/@material/textfield/_variables.scss 27:1 @forward node_modules/@material/textfield/_index.scss 1:1 @use node_modules/@angular/material/core/tokens/m2/mat/_form-field.scss 3:1 @use node_modules/@angular/material/core/tokens/m2/_index.scss 20:1 @use node_modules/@angular/material/core/tokens/_m3-tokens.scss 6:1 @use node_modules/@angular/material/core/theming/_definition.scss 6:1 @forward node_modules/@angular/material/_index.scss 7:1 @use src/app/module/homepage/homepage.component.scss 2:1 root stylesheet
bmarcotte
commented
2 months ago While poking around the Material repo yesterday, I ran across a ticket, angular/components#26603, from over a year ago, describing very similar errors. There's a comment from a user named @Carniatto, who posted a link to a repo he created to reproduce this issue. I made a fork of that repo, and added on some of the script and Docker pieces that I mentioned here previously. I ran 100 batches of 2 builds each (two different Node versions), and saw 11 failures total. Although that's not as high of a failure rate as we've seen we our larger proprietary code builds, at least we now have a publicly accessible repo that can reproduce the issue:
https://github.com/bmarcotte/sass-import-issue-repro
Is this enough for the Angular team to start investigating the issue, @alan-agius4?
Carniatto
commented
2 months ago While poking around the Material repo yesterday, I ran across a ticket, angular/components#26603, from over a year ago, describing very similar errors. There's a comment from a user named @Carniatto, who posted a link to a repo he created to reproduce this issue. I made a fork of that repo, and added on some of the script and Docker pieces that I mentioned here previously. I ran 100 batches of 2 builds each (two different Node versions), and saw 11 failures total. Although that's not as high of a failure rate as we've seen we our larger proprietary code builds, at least we now have a publicly accessible repo that can reproduce the issue:
https://github.com/bmarcotte/sass-import-issue-repro
Is this enough for the Angular team to start investigating the issue, @alan-agius4?
Thanks for the mention
For us, the resolution was taking so long that we actually worked around this issue. We decided to make all our theming hardcores CSS variables.
This is not ideal since you cannot grab colors from the theme defined on Angular Material, but we needed to move on.
ptandler
commented
2 months ago We have changed our CI builds from using alpine to slim, and so far, the results are looking much better. We don't have any hypothesis yet as to why an alpine image would lead to this issue, or why it would only do so intermittently, but at this point, we have enough data to want to avoid them in our environment.
We tried this and switched to 22-slim - and so far it really looks much better - many thanks for sharing @bmarcotte!
alan-agius4
commented
2 months ago Thanks for the reproduction, @bmarcotte and @Carniatto. I ran the test four times with 20, 50, 100, and 200 builds, respectively. Out of a total of 370 builds, there were no failed builds.
$ uname -a
Linux XXXXXX 6.6.15-2rodete2-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.6.15-2rodete2 (2024-03-19) x86_64 GNU/Linux$ docker -v
Docker version 20.10.21, build baeda1f$ ./run-tests.sh 10
...
[Wed Jun 26 08:21:14 AM UTC 2024] All batches completed
[Wed Jun 26 08:21:14 AM UTC 2024] Total runs: 20, failed: 0$ ./run-tests.sh 25
...
[Wed Jun 26 09:40:30 AM UTC 2024] All batches completed
[Wed Jun 26 09:40:30 AM UTC 2024] Total runs: 50, failed: 0$ ./run-tests.sh 50
...
[Wed Jun 26 01:12:43 PM UTC 2024] All batches completed
[Wed Jun 26 01:12:43 PM UTC 2024] Total runs: 100, failed: 0$ ./run-tests.sh 100
...
[Wed Jun 26 03:53:34 PM UTC 2024] All batches completed
[Wed Jun 26 03:53:34 PM UTC 2024] Total runs: 200, failed: 0While I can't completely rule out a bug in the Angular CLI, it's highly unlikely. Additionally, this issue seems to occur only with Docker, which the CLI doesn't have any special logic for. Module resolution relies on the location and availability of files in the system. If the file structure, availability, and access on disk remain consistent, so should the resolution across different builds.
The intermittent nature of the problem suggests it's not an Angular CLI bug, as the same code path is used for all builds. Builds are hermetic, and the same input should consistently produce the same output.
bmarcotte
commented
2 months ago @alan-agius4 Do you happen to know how many CPUs your Docker environment is configured to use? Most of my previous testing was done with a limit of 4 cpus. In some further testing today, I noticed that if I set that lower, to 1 or 2 cpus, the error rate increased significantly (sometimes over 50%), and if I set it higher to 8, the failures basically disappeared.
Note that I only saw that difference when changing it via Resource Allocation > CPU limit in the Settings tab of Docker Desktop (for Mac), which I think sets a limit for the entire Docker engine. Setting a cpu resource limit per container, via docker-compose.yaml (i.e. deploy.resources.limits.cpus=xxx) did not seem to have an effect (other than run time of the build).
If that observation is also seen by others, perhaps sass-loader, or something else in the sass toolchain, is hitting a timeout threshold under high CPU usage that it doesn't encounter when spread out over more cores?
alan-agius4
commented
2 months ago @bmarcotte, that's a great observation. By default, the CLI uses a maximum of 4 workers because, in the past, there wasn't a reliable way to determine the actual available CPUs within containers.
However, Node.js recently introduced a new API that provides more accurate values when used within containers (see this pull request). This feature is available starting from version 18.0.5. Could you try updating to the latest patch and see if the issue persists?
You can also limit the number of workers by setting the environment variable NG_BUILD_MAX_WORKERS=<number>.
In my environment, Docker can use 24 CPUs.
robertIsaac
commented
2 months ago @bmarcotte I noticed the same here, setting it to 2 CPU make the rate more than 50%, I actually mentioned this before https://github.com/angular/angular-cli/issues/27167#issuecomment-2163001161
@alan-agius4 did you try to limit your docker for 2/3/4 CPUs to test?
update: using Azure configuration of 2 CPU and 7 GB now I have a similar error rate Total runs: 20, failed: 4
foto-andreas
commented
2 months ago We see the bugs on a root server with 12 CPUs without Docker. Ubuntu 22.04
alan-agius4
commented
2 months ago @robertIsaac / @bmarcotte, I reduced Docker's CPU limit to 2 and did encountered the failures.
Here’s how I resolved them:
Configured NG_BUILD_MAX_WORKERS=1 to restrict Angular CLI to a single worker. This adjustment accounts for available CPUs minus one. It's necessary for Angular CLI versions prior to 18.0.5 (details in this comment).
In the meantime I have also created a PR to update the logic introduced in 18.0.5 to reduce the maximum number of workers to the available CPUs minus 1.
Note: Enabling NX parallelism should be carefully managed to prevent resource contention during builds. Personally, I suggest avoiding parallel builds unless there are at least 4 CPUs available.
robertIsaac
commented
2 months ago Hi @alan-agius4 We don't use parallel build in our project
alan-agius4
commented
2 months ago @robertIsaac, are you still setting NG_BUILD_MAX_WORKERS? If you're using Angular versions below 18, it's necessary to configure this, otherwise each build will default to using up to 4 workers.
robertIsaac
commented
2 months ago We don't, I will setup it today and monitor the pipeline Will leave another comment if it doesn't work
Uschi003
commented
2 months ago Unfortunately, we still got the issue after setting NG_BUILD_MAX_WORKERS = 1 using Angular 17.3. We still are using NX parallelism, as our CI pipeline has 10 CPUs available.
alan-agius4
commented
2 months ago @Uschi003, Are you also using for CI is it Docker? What is the maximum number of parallel builds you have configured?
Uschi003
commented
2 months ago we are using jenkins for CI on Ubuntu 22.04 without docker. we have configured a maximum of 12 parallel builds for nx in a test-pipeline to reproduce the issue although we only have 9 projects to build (9 apps and 3 libs). the error can also occur in our tests using karma.
alan-agius4
commented
2 months ago If you have 10 CPU cores and are building 9 applications in parallel, you'll need more than 10 cores to avoid resource contention.
Consider this: setting NG_BUILD_MAX_WORKERS to 1 forces each application build to use a single worker. However, each build also utilizes an additional main thread, totaling 2 threads per application.
Furthermore, building numerous applications simultaneously will likely strain file access operations.
I would start by reducing the number of parallel builds to find an optimal setting. Personally, I wouldn't exceed 4 parallel builds, but you could try 5 and monitor the impact on file operations.
wvankuipers
commented
1 month ago I like how this thread is finally getting to the core of the issue. (after the prev. thread that got closed was stale for several months) Thank you @alan-agius4 for the response.
I would start by reducing the number of parallel builds to find an optimal setting. Personally, I wouldn't exceed 4 parallel builds, but you could try 5 and monitor the impact on file operations.
Unfortunatly that's not a very sustainable solution as you might not always have control over the number of builds that run on a server. (like running with Github actions for example)
alan-agius4
commented
1 month ago You're absolutely right, @wvankuipers. In the above case, when configuring parallel builds within a single workflow using NX you can configure the number of builds that are ran in parallel within a single workflow.
it's always crucial to consider the overall workload on the runner server. This includes not just GitHub Actions workflows but also any other processes running on the machine.
For instance, if your physical server has 10 CPUs and you execute 4 workflows at the same time, each with 2 parallel builds, you will likely overload the system and encounter build failures.
That said, the underlying VM often does a reasonable job of dynamically allocating resources.
In simpler terms, be mindful of potential workload when creating workflows with parallel tasks.
Uschi003
commented
1 month ago Thank you for your time, @alan-agius4, in looking into this issue.
I have reduced NX parallelism to 4 and set NG_BUILD_MAX_WORKERS to 1, but we are still seeing occasional exceptions during our builds (although it seems that they are less likely to appear after a short amount of testing). In our pipeline, we are also building multiple Gradle projects simultaneously, so there is no doubt that we have a high load on the server. Disabling NX parallelism altogether would have a noticeable impact on our build times, but we can try and monitor this.
Although I'm not quite certain that this is the root cause of the issue, what I don't understand is why we did not have these flaky build failures when we were still using Angular 14, also built with NX in parallel.
alan-agius4
commented
1 month ago The main difference since Angular 14 is that the Angular CLI started utilizing worker threads which become stable.
Previously, many CPU-intensive tasks, such as Sass compilations and JS and CSS minification, were performed on the main thread, which could cause performance bottlenecks.
robertIsaac
commented
1 month ago hi @alan-agius4
it failed again, with NG_BUILD_MAX_WORKERS: 1 and no nx parallel builds
it was with nx serve not nx build, does @angular-devkit/build-angular:dev-server uses NG_BUILD_MAX_WORKERS as well?
note: we use nx serve in our pipeline as part of the e2e testing
alan-agius4
commented
1 month ago Yes it does, do you use the Vite based dev-server or the webpack based?
robertIsaac
commented
1 month ago Vite based dev-server
robertIsaac
commented
1 month ago my observation over the past week is that now it never fail on build in the past it was happening more frequently in the build than in the serve, so for some reason limiting the worker only work for production build
alan-agius4
commented
1 month ago I wonder if this issue is caused by Vite dependency optimization and caching, which are disabled by default in CI environments. Can you confirm whether your CI environment is setting a CI environment variable?
robertIsaac
commented
1 month ago I set the variable incorrectly in the yml file for the test I fixed it, and will merge it tomorrow and monitor the pipeline
Yberion
commented
1 month ago Hello @alan-agius4 I don't know if it's the same issue, but I'm getting the same problem with nx.
I've made a ticket on nx with a reproduction, maybe this could help for the investigation.
Uschi003
commented
1 month ago we have set the environment variable CI in our environment and are still seeing these build failures. Also we have previously switched from webpack to esbuild but the build failures appeared in both build systems so it does not seem like a caching issue in vite to me.
Command
build
Is this a regression?
The previous version in which this bug was not present was
No response
Description
about 5% of our pipeline fail with this error
it happened only one with
ng serveand it was very difficult to resolve, my colleague had to uninstall node and install it again previously he tried to delete the project and clone it again, remove yarn cache but neither workMinimal Reproduction
it's impossible to reproduce, since even rerunning the same one succeed
it's not code error
this happens even when we were using webpack (we are using the application builder which uses esbuild), I noticed it in versions 15 and 17 of Angular
Exception or Error
Your Environment
Anything else relevant?
No response