tinesoft / nxrocks

Set of Nx plugins to enhance your Nx workspace (even more!)
MIT License
309 stars 36 forks source link

[Question] Maven dependency management for monorepo libs #204

Open ToppScorer opened 11 months ago

ToppScorer commented 11 months ago

Hi!

I'm not sure if I use nx-spring-boot in the correct way:

I have a monorepo where I have a Maven library project (under libs) and a Maven application project (under apps). In the POM of the application a dependency to the library project is defined:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>test</groupId>
  <artifactId>test-service</artifactId>
  <version>${revision}</version>
  <dependencies>
    <dependency>
      <groupId>test</groupId>
      <artifactId>test-library</artifactId>
      <version>${project.version}</version>
    </dependency>
  </dependencies>
  ...

When I try to install the application, I would expect, that the library is installed first:

    "install": {
      "executor": "@nxrocks/nx-spring-boot:install",
      "options": {
        "root": "apps/test-service"
      },
      "dependsOn": [
        "^install",
        "clean"
      ],
      "outputs": [
        "{workspaceRoot}/apps/test-service/target"
      ]
    }

But the Maven build fails because it cannot find the library in the defined Maven repositories. It seems, that the POM is not analyzed.

tinesoft commented 11 months ago

Hi @ToppScorer

Thanks for using the plugin and for reporting this.

Can you provide the following information,

Your setup seems correct.

It possible that the mvn install on the library failed (because a Spring Boot library is not supposed to have a @SpringBootApplication nor @SpringBootTest, and right now, the default generated library does).

ToppScorer commented 11 months ago

nx report

 Node   : 16.15.0
 OS     : darwin-x64
 npm    : 8.10.0

 nx (global)    : 17.1.2
 nx             : 17.1.2
 @nx/js         : 17.1.2
 @nx/workspace  : 17.1.2
 @nx/devkit     : 17.1.2
 @nrwl/tao      : 17.1.2
 ---------------------------------------
 Community plugins:
 @nxrocks/nx-spring-boot : 9.0.2

Stacktrace

nx install test-service

> nx run test-service:install

Executing command: ./mvnw install
[INFO] Scanning for projects...
[INFO]
[INFO] ----------------< test:test-service >-----------------
[INFO] Building test-service HEAD-SNAPSHOT
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[WARNING] The POM for test:test-library:jar:HEAD-SNAPSHOT is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.161 s
[INFO] Finished at: 2023-11-20T14:51:12+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project test-service: Could not resolve dependencies for project test:test-service:jar:HEAD-SNAPSHOT: The following artifacts could not be resolved: test:test-library:jar:HEAD-SNAPSHOT (absent): test:test-library:jar:HEAD-SNAPSHOT was not found in https://gitlab.com/api/v4/groups/.../-/packages/maven during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of ... has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
Failed to execute command: ./mvnw install
Error: Command failed: ./mvnw install
    at checkExecSyncError (node:child_process:828:11)
    at execSync (node:child_process:899:15)
    at runBuilderCommand (/test-monorepo/node_modules/@nxrocks/common-jvm/src/lib/utils/utils.js:22:38)
    at runBootPluginCommand (/test-monorepo/node_modules/@nxrocks/nx-spring-boot/src/utils/boot-utils.js:19:47)
    at /test-monorepo/node_modules/@nxrocks/nx-spring-boot/src/executors/install/executor.js:10:54
    at Generator.next (<anonymous>)
    at /test-monorepo/node_modules/tslib/tslib.js:169:75
    at new Promise (<anonymous>)
    at Object.__awaiter (/test-monorepo/node_modules/tslib/tslib.js:165:16)
    at installExecutor (/test-monorepo/node_modules/@nxrocks/nx-spring-boot/src/executors/install/executor.js:8:20)
    at runExecutorInternal (/test-monorepo/node_modules/nx/src/command-line/run/run.js:73:19)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async /test-monorepo/node_modules/nx/src/command-line/run/run.js:154:44
    at async handleErrors (/test-monorepo/node_modules/nx/src/utils/params.js:9:16)
    at async process.<anonymous> (/test-monorepo/node_modules/nx/bin/run-executor.js:59:28)

 ————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Ran target install for project test-service (4s)

    ✖    1/1 failed
    ✔    0/1 succeeded [0 read from cache]

As I'm using the Maven CI Friendly Versions approach, the dependency resolution should work.

I think the main problem is that I assume that the library is installed automatically, based on the information from the POM. The library has not been installed manually before.

tinesoft commented 11 months ago

Hi,

I was not aware of that Maven CI friendly versioning...

But I did try to reproduce your setup (one app, one lib, app's pom.xml has a dependency on lib, using a simple version like 0.0.1-SNAPSHOT) and I confirm that the library is built/installed prior to the app when you do run nx install [app-name].

See below:

npx nx install bootapp --skip-nx-cache --verbose

 >  NX   Running target install for project bootapp and 1 task it depends on:

 —————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

> nx run bootlib:install

Executing command: ./mvnw install 
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------< com.example:demo-lib >------------------------
[INFO] Building bootlib 0.0.1-SNAPSHOT
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- resources:3.3.1:resources (default-resources) @ demo-lib ---
[INFO] Copying 1 resource from src/main/resources to target/classes
[INFO] Copying 0 resource from src/main/resources to target/classes
[INFO] 
[INFO] --- compiler:3.11.0:compile (default-compile) @ demo-lib ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- resources:3.3.1:testResources (default-testResources) @ demo-lib ---
[INFO] skip non existing resourceDirectory /Users/tine/Documents/angular/nx-boot/bootlib/src/test/resources
[INFO] 
[INFO] --- compiler:3.11.0:testCompile (default-testCompile) @ demo-lib ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- surefire:3.0.0:test (default-test) @ demo-lib ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.example.demo.BootlibApplicationTests
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.038 s - in com.example.demo.BootlibApplicationTests
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] --- jar:3.3.0:jar (default-jar) @ demo-lib ---
[INFO] 
[INFO] --- install:3.1.1:install (default-install) @ demo-lib ---
[INFO] Installing /Users/tine/Documents/angular/nx-boot/bootlib/pom.xml to /Users/tine/.m2/repository/com/example/demo-lib/0.0.1-SNAPSHOT/demo-lib-0.0.1-SNAPSHOT.pom
[INFO] Installing /Users/tine/Documents/angular/nx-boot/bootlib/target/demo-lib-0.0.1-SNAPSHOT.jar to /Users/tine/.m2/repository/com/example/demo-lib/0.0.1-SNAPSHOT/demo-lib-0.0.1-SNAPSHOT.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.636 s
[INFO] Finished at: 2023-11-21T09:52:38+01:00
[INFO] ------------------------------------------------------------------------

> nx run bootapp:install

Executing command: ./mvnw install 
[INFO] Scanning for projects...
[INFO] 
[INFO] --------------------------< com.example:demo >--------------------------
[INFO] Building bootapp 0.0.1-SNAPSHOT
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- resources:3.3.1:resources (default-resources) @ demo ---
[INFO] Copying 1 resource from src/main/resources to target/classes
[INFO] Copying 0 resource from src/main/resources to target/classes
[INFO] 
[INFO] --- compiler:3.11.0:compile (default-compile) @ demo ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- resources:3.3.1:testResources (default-testResources) @ demo ---
[INFO] skip non existing resourceDirectory /Users/tine/Documents/angular/nx-boot/bootapp/src/test/resources
[INFO] 
[INFO] --- compiler:3.11.0:testCompile (default-testCompile) @ demo ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- surefire:3.0.0:test (default-test) @ demo ---
[INFO] Using auto detected provider org.apache.maven.surefire.junitplatform.JUnitPlatformProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.example.demo.BootappApplicationTests
09:52:42.480 [main] INFO org.springframework.test.context.support.AnnotationConfigContextLoaderUtils -- Could not detect default configuration classes for test class [com.example.demo.BootappApplicationTests]: BootappApplicationTests does not declare any static, non-private, non-final, nested classes annotated with @Configuration.
09:52:42.589 [main] INFO org.springframework.boot.test.context.SpringBootTestContextBootstrapper -- Found @SpringBootConfiguration com.example.demo.BootappApplication for test class com.example.demo.BootappApplicationTests
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.1.5)
2023-11-21T09:52:42.936+01:00  INFO 11994 --- [           main] c.example.demo.BootappApplicationTests   : Starting BootappApplicationTests using Java 21.0.1 with PID 11994 (started by tine in /Users/tine/Documents/angular/nx-boot/bootapp)
2023-11-21T09:52:42.937+01:00  INFO 11994 --- [           main] c.example.demo.BootappApplicationTests   : No active profile set, falling back to 1 default profile: "default"
2023-11-21T09:52:43.871+01:00  INFO 11994 --- [           main] c.example.demo.BootappApplicationTests   : Started BootappApplicationTests in 1.138 seconds (process running for 2.092)
WARNING: A Java agent has been loaded dynamically (/Users/tine/.m2/repository/net/bytebuddy/byte-buddy-agent/1.14.9/byte-buddy-agent-1.14.9.jar)
WARNING: If a serviceability tool is in use, please run with -XX:+EnableDynamicAgentLoading to hide this warning
WARNING: If a serviceability tool is not in use, please run with -Djdk.instrument.traceUsage for more information
WARNING: Dynamic loading of agents will be disallowed by default in a future release
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.487 s - in com.example.demo.BootappApplicationTests
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] --- jar:3.3.0:jar (default-jar) @ demo ---
[INFO] 
[INFO] --- spring-boot:3.1.5:repackage (repackage) @ demo ---
[INFO] Replacing main artifact /Users/tine/Documents/angular/nx-boot/bootapp/target/demo-0.0.1-SNAPSHOT.jar with repackaged archive, adding nested dependencies in BOOT-INF/.
[INFO] The original artifact has been renamed to /Users/tine/Documents/angular/nx-boot/bootapp/target/demo-0.0.1-SNAPSHOT.jar.original
[INFO] 
[INFO] --- install:3.1.1:install (default-install) @ demo ---
[INFO] Installing /Users/tine/Documents/angular/nx-boot/bootapp/pom.xml to /Users/tine/.m2/repository/com/example/demo/0.0.1-SNAPSHOT/demo-0.0.1-SNAPSHOT.pom
[INFO] Installing /Users/tine/Documents/angular/nx-boot/bootapp/target/demo-0.0.1-SNAPSHOT.jar to /Users/tine/.m2/repository/com/example/demo/0.0.1-SNAPSHOT/demo-0.0.1-SNAPSHOT.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.449 s
[INFO] Finished at: 2023-11-21T09:52:45+01:00
[INFO] ------------------------------------------------------------------------

 ——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target install for project bootapp and 1 task it depends on

Question to you: When running nx graph , can you confirm that app and lib project are connected? like shown below

image

ToppScorer commented 11 months ago

No, they are not connected, like in you example, so my configuration seems to be wrong. Can you show me your nx.json files or do you have that example uploaded?

tinesoft commented 11 months ago

No, they are not connected,

Ok that's your issue then! ^^

The setting to connect the install targets can be set either globally in the nx.json ( in the targetDefaults section) or locally in the project.json of the application project( this is my case):

project.json

{
  "name": "bootapp",
  "$schema": "../node_modules/nx/schemas/project-schema.json",
  "sourceRoot": "bootapp/src",
  "projectType": "application",
  "targets": {
    "build": {
      "executor": "@nxrocks/nx-spring-boot:build",
      "options": {
        "root": "./bootapp"
      },
      "dependsOn": [
        "^install"
      ],
      "outputs": [
        "{workspaceRoot}/bootapp/target"
      ]
    },
    "install": {
      "executor": "@nxrocks/nx-spring-boot:install",
      "options": {
        "root": "./bootapp"
      },
      "dependsOn": [
        "^install"
      ],
      "outputs": [
        "{workspaceRoot}/bootapp/target"
      ]
    },
    "test": {
      "executor": "@nxrocks/nx-spring-boot:test",
      "options": {
        "root": "./bootapp"
      },
      "outputs": [
        "{workspaceRoot}/bootapp/target"
      ]
    },
    "clean": {
      "executor": "@nxrocks/nx-spring-boot:clean",
      "options": {
        "root": "./bootapp"
      }
    },
    "run": {
      "executor": "@nxrocks/nx-spring-boot:run",
      "options": {
        "root": "./bootapp"
      }
    },
    "serve": {
      "executor": "@nxrocks/nx-spring-boot:serve",
      "options": {
        "root": "./bootapp"
      }
    },
    "build-image": {
      "executor": "@nxrocks/nx-spring-boot:build-image",
      "options": {
        "root": "./bootapp"
      },
      "outputs": [
        "{workspaceRoot}/bootapp/target"
      ]
    },
    "build-info": {
      "executor": "@nxrocks/nx-spring-boot:build-info",
      "options": {
        "root": "./bootapp"
      }
    }
  },
  "tags": []
}
ToppScorer commented 11 months ago

My project.json looks the same. I will create an example project from scratch, maybe it's because of the Maven CI Friendly approach.

tinesoft commented 11 months ago

Can share the two pom.xml and the two project.json files ?

ToppScorer commented 11 months ago

I think I found the problem. The pom.xml from my app looks like this now (create a new project from scratch compared to the first post):

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>com.example</groupId>
        <artifactId>example-app-parent</artifactId>
        <version>${revision}</version>
    </parent>
    <groupId>com.example</groupId>
    <artifactId>example-app</artifactId>
    <packaging>war</packaging>
    <description>Demo project for Spring Boot</description>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>com.example</groupId>
            <artifactId>example-lib</artifactId>
            <version>${project.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-tomcat</artifactId>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <image>
                        <builder>paketobuildpacks/builder-jammy-base:latest</builder>
                    </image>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

So, it has no <artifactId>spring-boot-starter-parent</artifactId> parent. When I add that fragment (even only as a comment):

<!--
    <artifactId>spring-boot-starter-parent</artifactId>
 -->

the example-lib will be build before, otherwise not at all.

What's the purpose behind that?

tinesoft commented 11 months ago

Humm... That's very odd...

there must be something else in your pom.xml that is messing the dependency detection system of the plugin. Only that section should matter:

        <dependency>
            <groupId>com.example</groupId>
            <artifactId>example-lib</artifactId>
            <version>${project.version}</version>
        </dependency>

Can you create a minimal reproduction project and push it to github ?

ToppScorer commented 11 months ago

Yes, here it is https://github.com/ToppScorer/test-monorepo

With that setup, running nx run example-app:install --skip-nx-cache will not build the example-lib (and also not the example-lib-parent, but that's maybe a different problem).

By changing the pom.xml for the example-app to have

<!--
  <artifactId>spring-boot-starter-parent</artifactId>
  -->

it will build the example-lib first.

ToppScorer commented 11 months ago

I found another problem, which is maybe related. If you don't specify the groupId in one of the pom.xml because it's inherited from the parent, there are also problems resolving dependent libs.

tinesoft commented 11 months ago

Thanks for the repro project. I can see your problem now:

You didn't create the app and library projects right:

You created them as Maven multi-module projects while using them as simple maven projects.

Either you create 1 true Maven parent module (eg: example-parent) with 2 children modules underneath (eg example-app and example-lib)

├── example-parent
│       ├── example-app
│       │    ├── pom.xml
│       │    ├── project.json
│       │    └── src/
│       ├── example-lib
│       │    ├── pom.xml
│       │    ├── project.json
│       │    └── src/
│       ├── mvnw
│       ├── mvnw.cmd
│       ├── pom.xml
│       ├── project.json
│       └── target
└── nx.json

Or, you create two individual maven projects (like I did below)


├── example-app
│    ├── pom.xml
│    ├── project.json
│    ├── mvnw
│    ├── mvnw.cmd
│    └── src/
├── example-lib
│    ├── pom.xml
│    ├── project.json
│    ├── mvnw
│    ├── mvnw.cmd
│    └── src/
└── nx.json
image
ToppScorer commented 11 months ago

You didn't create the app and library projects right

The example repo has a perfectly valid Maven setup. The only real difference I can see compared to your second example is, that the parent definitions are different (where the defined parent also has the spring-boot-starter-parent as a parent). In fact, it is almost running, but it expects to have this spring-boot-starter-parent fragment and explicitly defined groupIds.

The second problem should be easy to fix. For the first one I don't really understand, why it has to have this exact parent. There are different ways, how a Maven Spring-Boot project setup can look like, e. g. importing the spring-boot-dependencies BOM. So, it would be great, if this plugin could also cover that.

tinesoft commented 11 months ago

Hi,

The example repo has a perfectly valid Maven setup.

Yes you're right (bad wording from my part), taken individually, each of the project is indeed a valid Maven project What I really meant to say, is that, the way you are using those maven spring boot projects, is not really how I have envisioned nor seen it in most popular projects, so I implemented the multi module support in a certain manner that doesn't match your use case...

The second problem should be easy to fix. For the first one I don't really understand, why it has to have this exact parent.

Right now, the plugin rely on <artifactId>spring-boot-starter-parent</artifactId> to identify which projects in the Nx workspace, are Spring boot projects. Only those are considered for the dependency graph and related features.

I will have another look into this, and try to provide a fix for your use case too

Stay tuned

ToppScorer commented 11 months ago

I'm not sure if it's related, but when I change the spring-boot-starter-parent version in the example-lib-parent, e.g. from 3.1.5 to 3.1.6, and all libs and apps were built and cached before, and I'm running the same build-image command, then everything will be read from the cache instead of building everything new.

ToppScorer commented 11 months ago

Could it be, that the dependency graph is wrong:

image

I would have expected, that the parents are dependencies of their childs and not the other way around, which in this case would mean, that e.g. example-app-parent is a dependency of example-app as the parent has to be build first.