[Bug] Error When Running openchallenges-build-images Post node_module Directory Deletion

mdsage1 commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

What product(s) are you seeing the problem on?

OpenChallenges, Sage Monorepo

Current behavior

Rebuild dev container post rebase operations
Deleted the node_modules folder by running rm -rf /workspaces/sage-monorepo/node_modules
Run workspace-install to bring in the required components lost from the above folder deletion
Attempt to build images by running openchallenges-build-images and receive this error message `> NX Ran target build-image for 26 projects and 23 tasks they depend on (5m)

✔ 45/47 succeeded [0 read from cache]

✖ 2/47 targets failed, including the following:
- nx run openchallenges-challenge-service:build-image-base
- nx run openchallenges-image-service:build-image-base`
  1. Additional error image information: BUILD FAILED in 1m 7s 1 actionable task: 1 executed Failed to execute command: ./gradlew bootBuildImage Error: Command failed: ./gradlew bootBuildImage at checkExecSyncError (node:child_process:885:11) at execSync (node:child_process:957:15) at runBuilderCommand (/workspaces/sage-monorepo/node_modules/@nxrocks/common/src/lib/core/jvm/utils.js:20:38) at runBootPluginCommand (/workspaces/sage-monorepo/node_modules/@nxrocks/nx-spring-boot/src/utils/boot-utils.js:18:43) at /workspaces/sage-monorepo/node_modules/@nxrocks/nx-spring-boot/src/executors/build-image/executor.js:10:54 at Generator.next (<anonymous>) at /workspaces/sage-monorepo/node_modules/tslib/tslib.js:118:75 at new Promise (<anonymous>) at Object.__awaiter (/workspaces/sage-monorepo/node_modules/tslib/tslib.js:114:16) at buildImageExecutor (/workspaces/sage-monorepo/node_modules/@nxrocks/nx-spring-boot/src/executors/build-image/executor.js:8:20) at /workspaces/sage-monorepo/node_modules/nx/src/command-line/run/run.js:81:23 at Generator.next (<anonymous>) at fulfilled (/workspaces/sage-monorepo/node_modules/nx/node_modules/tslib/tslib.js:166:62) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) My EC2 instance has 32G of available memory and all actions performed on the instance.

Expected behavior

I would expect all docker images to be built following running openchallenges-build-images

Anything else?

No response

Commit ID

No response

Are you developing inside the dev container?

[X] I am using the dev container

Code of Conduct

[X] I agree to follow this project's Code of Conduct

tschaffter commented 1 year ago

@mdsage1 What branch / commit ID are you using? This information is needed to make sure we are using the same code.

tschaffter commented 1 year ago

Deleted the node_modules folder by running Run workspace-install to bring in the required components lost from the above folder deletion

Why should node_modules be removed?

Attempt to build images by running openchallenges-build-images and receive this error message

The logs show that these tasks fail:

nx run openchallenges-challenge-service:build-image-base
nx run openchallenges-image-service:build-image-base

Can you please try to run these command one at a time? Do the logs provide additional feedback?

Additional error image information:

The logs in this error message show that one command that fail is ./gradlew bootBuildImage. Try to run this command from within the folder of one of the projects that fail. E.g. apps/openchallenges/image-service.

None of the logs shared provide a clear answer of WHAT is failing. This is in part because the original error messages don't seem to be propagated up to Nx. The method to learned more about the issue is hinted above. Ultimately, running the command ./gradlew bootBuildImage from the folder apps/openchallenges/image-service should reveal the original error message.

mdsage1 commented 1 year ago

I was still on the main branch

tschaffter commented 1 year ago

I was still on the main branch

FYI I know you have updated your main branch today because we did it together. Ideally, the commit ID should be provided to unambiguously identify the version of the code used.

I will try to run the above command on my side.

tschaffter commented 1 year ago

Here is the commit ID of my main branch:

$ git rev-parse HEAD
80cb1420e521b546ac0f10b3d729ec46c296d2e3

However, I can't reproduce the issue:

$ openchallenges-build-images 

 >  NX   The following projects do not have a configuration for any of the provided targets ("build-image")

   - openchallenges-api-client-angular-deprecated
   - openchallenges-api-client-angular
   - openchallenges-challenge-search
   - openchallenges-app-config-data
   - openchallenges-api-description
   - openchallenges-kafka-consumer
   - openchallenges-kafka-producer
   - openchallenges-api-client-r
   - openchallenges-user-profile
   - openchallenges-kafka-model
   - openchallenges-kafka-admin
   - openchallenges-org-profile
   - openchallenges-org-search
   - openchallenges-challenge
   - openchallenges-not-found
   - openchallenges-notebook
   - openchallenges-assets
   - openchallenges-themes
   - openchallenges-signup
   - openchallenges-styles
   - openchallenges-config
   - openchallenges-about
   - openchallenges-kafka
   - openchallenges-infra
   - openchallenges-home
   - openchallenges-team
   - openchallenges-util
   - openchallenges-ui

    ✔  nx run openchallenges-api-client-angular-deprecated:build:production  [local cache]
    ✔  nx run openchallenges-api-description:build-individuals (2s)
    ✔  nx run openchallenges-api-description:build (1s)
    ✔  nx run openchallenges-api-client-angular:build:production (6s)
    ✔  nx run shared-java-util:build (14s)
    ✔  nx run openchallenges-app-config-data:build (17s)
    ✔  nx run shared-java-util:install (3s)
    ✔  nx run openchallenges-app-config-data:install (3s)
    ✔  nx run openchallenges-kafka-model:build (14s)
    ✔  nx run openchallenges-kafka-admin:build (5s)
    ✔  nx run openchallenges-kafka-model:install (4s)
    ✔  nx run openchallenges-kafka-admin:install (4s)
    ✔  nx run openchallenges-service-registry:build-image-base (25s)
    ✔  nx run openchallenges-api-gateway:build-image-base (24s)
    ✔  nx run openchallenges-kafka-consumer:build (6s)
    ✔  nx run openchallenges-kafka-producer:build (7s)
    ✔  nx run openchallenges-kafka-consumer:install (3s)
    ✔  nx run openchallenges-app:build:production (45s)
    ✔  nx run openchallenges-kafka-producer:install (4s)
    ✔  nx run openchallenges-organization-service:build-image-base (31s)
    ✔  nx run openchallenges-challenge-service:build-image-base (33s)
    ✔  nx run openchallenges-keycloak:build-image (2s)
    ✔  nx run openchallenges-service-registry:build-image (2s)
    ✔  nx run openchallenges-image-service:build-image-base (37s)
    ✔  nx run openchallenges-mariadb:build-image (3s)
    ✔  nx run openchallenges-config-server:build-image-base (11s)
    ✔  nx run openchallenges-mysqld-exporter:build-image (2s)
    ✔  nx run openchallenges-elasticsearch:build-image (3s)
    ✔  nx run openchallenges-prometheus:build-image (2s)
    ✔  nx run openchallenges-api-docs:build-image (2s)
    ✔  nx run openchallenges-postgres:build-image (2s)
    ✔  nx run openchallenges-thumbor:build-image (3s)
    ✔  nx run openchallenges-rstudio:build-image (3s)
    ✔  nx run openchallenges-grafana:build-image (2s)
    ✔  nx run openchallenges-mongo:build-image (2s)
    ✔  nx run openchallenges-zipkin:build-image (2s)
    ✔  nx run openchallenges-api-gateway:build-image (17s)
    ✔  nx run openchallenges-vault:build-image (3s)
    ✔  nx run openchallenges-apex:build-image (3s)
    ✔  nx run openchallenges-user-service:build-image (17s)
    ✔  nx run openchallenges-auth-service:build-image (26s)
    ✔  nx run openchallenges-core-service:build-image (26s)
    ✔  nx run openchallenges-app:build-image (4s)
    ✔  nx run openchallenges-organization-service:build-image (2s)
    ✔  nx run openchallenges-challenge-to-elasticsearch-service:build-image (17s)
    ✔  nx run openchallenges-kaggle-to-kafka-service:build-image (13s)
    ✔  nx run openchallenges-config-server:build-image (1s)
    ✔  nx run openchallenges-challenge-service:build-image (14s)
    ✔  nx run openchallenges-image-service:build-image (14s)

 ———————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target build-image for 26 projects and 23 tasks they depend on (3m)

@mdsage1 Please share the information requested here. Only after doing so, try to clean the build folder of the projects that failed.

nx clean openchallenges-challenge-service
nx clean openchallenges-image-service

Then try again to build the base images:

nx build-image-base openchallenges-challenge-service
nx build-image-base openchallenges-image-service

mdsage1 commented 1 year ago

@mdsage1 What branch / commit ID are you using? This information is needed to make sure we are using the same code. 33d2028cad8925118d2fbee89d12eab33376f715

mdsage1 commented 1 year ago

Deleted the node_modules folder by running Run workspace-install to bring in the required components lost from the above folder deletion

Why should node_modules be removed?

The node modules were causing another error when attempting to build all images by running openchallenges-build-images

Attempt to build images by running openchallenges-build-images and receive this error message

The logs show that these tasks fail:

nx run openchallenges-challenge-service:build-image-base

Output:

NX   Successfully ran target build-image-base for project openchallenges-challenge-service and 4 tasks it depends on (39s)

   Nx read the output from the cache instead of running the command for 2 out of 5 tasks.

   View logs and investigate cache misses at https://cloud.nx.app/runs/W1Ifq8KvHO

nx run openchallenges-image-service:build-image-base

Output:

 >  NX   Successfully ran target build-image-base for project openchallenges-image-service and 4 tasks it depends on (33s)

   Nx read the output from the cache instead of running the command for 2 out of 5 tasks.

   View logs and investigate cache misses at https://cloud.nx.app/runs/bnPxUg6nTt

Can you please try to run these command one at a time? Do the logs provide additional feedback?

Additional error image information:
Successfully built image 'ghcr.io/sage-bionetworks/openchallenges-image-service-base:local'
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.3/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
The logs in this error message show that one command that fail is ./gradlew bootBuildImage. Try to run this command from within the folder of one of the projects that fail. E.g. apps/openchallenges/image-service.

None of the logs shared provide a clear answer of WHAT is failing. This is in part because the original error messages don't seem to be propagated up to Nx. The method to learned more about the issue is hinted above. Ultimately, running the command ./gradlew bootBuildImage from the folder apps/openchallenges/image-service should reveal the original error message.

mdsage1 commented 1 year ago

The issue disappeared and we couldn't reproduce it.

Sage-Bionetworks / sage-monorepo