Multi-Module Maven Project

[sweetodev]

With the following configuration it works fine in a single project, unable to make it work in a multi modular maven project.

I've my architecture as follows.

Two Servers moduleA/src/main/resources/fileA.proto

moduleB/src/main/resources/fileB.proto

Client (Browser facing server end point calling above two servers) service/src/main

I specified following POM in moduleA to begin with but it's not generating stubs, please advise on how to approach a multi-modular project?

Dependencies

<dependencies>
    <dependency>
        <groupId>com.google.protobuf</groupId>
        <artifactId>protobuf-java</artifactId>
        <version>3.6.1</version>
    </dependency>
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-netty-shaded</artifactId>
        <version>1.15.1</version>
    </dependency>
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-protobuf</artifactId>
        <version>1.15.1</version>
    </dependency>
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-stub</artifactId>
        <version>1.15.1</version>
    </dependency>
</dependencies>

Build

<build>
    <extensions>
        <extension>
            <groupId>kr.motd.maven</groupId>
            <artifactId>os-maven-plugin</artifactId>
            <version>1.5.0.Final</version>
        </extension>
    </extensions>
    <finalName>Department</finalName>
    <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
        <plugins>
            <plugin>
                <artifactId>maven-clean-plugin</artifactId>
                <version>3.1.0</version>
            </plugin>
            <!-- see http://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_war_packaging -->
            <plugin>
                <artifactId>maven-resources-plugin</artifactId>
                <version>3.0.2</version>
            </plugin>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.0</version>
            </plugin>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
            </plugin>
            <plugin>
                <artifactId>maven-war-plugin</artifactId>
                <version>3.2.2</version>
            </plugin>
            <plugin>
                <artifactId>maven-install-plugin</artifactId>
                <version>2.5.2</version>
            </plugin>
            <plugin>
                <artifactId>maven-deploy-plugin</artifactId>
                <version>2.8.2</version>
            </plugin>

            <plugin>
                <groupId>org.xolstice.maven.plugins</groupId>
                <artifactId>protobuf-maven-plugin</artifactId>
                <version>0.5.1</version>
                <configuration>
                    <protocArtifact>com.google.protobuf:protoc:3.9.0:exe:${os.detected.classifier}</protocArtifact>
                    <pluginId>grpc-java</pluginId>
                    <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.22.1:exe:${os.detected.classifier}</pluginArtifact>
                </configuration>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>compile-custom</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
    </pluginManagement>
</build>

[sergei-ivanov]

Hi,

You need to move the configuration and executions sections for protobuf-maven-plugin into the plugins section of the build. The above configuration works for maven-compiler-plugin, because it is part of the default maven lifecycle. But protobuf-maven-plugin is not part of the default lifecycle, and that is why you need to specify the executions.

Also, I noticed that you are using protobuf-java:3.6.1 while using protoc:3.9.0. Ideally you should use the same version for both (you can extract it into a build property). Otherwise the generated classes may not be compatible with the library.

So, I have tried to fix if for you below:

<properties>
    <protobufVersion>3.9.0</protobufVersion>
    <grpcVersion>1.22.1</grpcVersion>
</properties>

<dependencies>
    <dependency>
        <groupId>com.google.protobuf</groupId>
        <artifactId>protobuf-java</artifactId>
        <version>${protobufVersion}</version>
    </dependency>
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-protobuf</artifactId>
        <version>${grpcVersion}</version>
    </dependency>
    ...
</dependencies>

<build>
    <extensions>
        <extension>
            <groupId>kr.motd.maven</groupId>
            <artifactId>os-maven-plugin</artifactId>
            <version>1.5.0.Final</version>
        </extension>
    </extensions>
    <finalName>Department</finalName>
    <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
        <plugins>
            ...
            <plugin>
                <groupId>org.xolstice.maven.plugins</groupId>
                <artifactId>protobuf-maven-plugin</artifactId>
                <version>0.5.1</version>
            </plugin>
            ...
        </plugins>
    </pluginManagement>

    <plugins>
        ...
        <plugin>
            <groupId>org.xolstice.maven.plugins</groupId>
            <artifactId>protobuf-maven-plugin</artifactId>
            <configuration>
                <protocArtifact>com.google.protobuf:protoc:${protobufVersion}:exe:${os.detected.classifier}</protocArtifact>
                <pluginId>grpc-java</pluginId>
                <pluginArtifact>io.grpc:protoc-gen-grpc-java:${grpcVersion}:exe:${os.detected.classifier}</pluginArtifact>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>compile</goal>
                        <goal>compile-custom</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        ...
    </plugins>
</build>

[sweetodev]

Thanks @sergei-ivanov for your input. I've updated submodule POM. I'm writing under both my parent and submodule POM for your review. What mvn command should I trigger to build all submodules while generating stubs. (I'm getting success with mvn compile but without stub generation)

Parent

<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>Demo</artifactId>
<packaging>pom</packaging>
<version>1.0-SNAPSHOT</version>
<modules>
    <module>ModuleA</module>
    <module>ModuleB</module>
    <module>RESTService</module>
</modules>
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>2.3.2</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
    </plugins>
</build>

ModuleA

<parent>
    <artifactId>Demo</artifactId>
    <groupId>com.example</groupId>
    <version>1.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>ModuleA</artifactId>
<packaging>jar</packaging>
<name>Module A</name>
<url>http://www.example.com</url>

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
    <protobufVersion>3.9.0</protobufVersion>
    <grpcVersion>1.22.1</grpcVersion>
</properties>

<dependencies>
    <dependency>
        <groupId>com.google.protobuf</groupId>
        <artifactId>protobuf-java</artifactId>
        <version>${protobufVersion}</version>
    </dependency>
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-netty-shaded</artifactId>
        <version>1.15.1</version>
    </dependency>
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-protobuf</artifactId>
        <version>${grpcVersion}</version>
    </dependency>
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-stub</artifactId>
        <version>1.15.1</version>
    </dependency>
</dependencies>

<build>
    <extensions>
        <extension>
            <groupId>kr.motd.maven</groupId>
            <artifactId>os-maven-plugin</artifactId>
            <version>1.5.0.Final</version>
        </extension>
    </extensions>
    <finalName>ModuleA</finalName>
    <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) -->
        <plugins>
            <plugin>
                <artifactId>maven-clean-plugin</artifactId>
                <version>3.1.0</version>
            </plugin>
            <!-- see http://maven.apache.org/ref/current/maven-core/default-bindings.html#Plugin_bindings_for_war_packaging -->
            <plugin>
                <artifactId>maven-resources-plugin</artifactId>
                <version>3.0.2</version>
            </plugin>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.0</version>
            </plugin>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
            </plugin>
            <plugin>
                <artifactId>maven-war-plugin</artifactId>
                <version>3.2.2</version>
            </plugin>
            <plugin>
                <artifactId>maven-install-plugin</artifactId>
                <version>2.5.2</version>
            </plugin>
            <plugin>
                <artifactId>maven-deploy-plugin</artifactId>
                <version>2.8.2</version>
            </plugin>

            <plugin>
                <groupId>org.xolstice.maven.plugins</groupId>
                <artifactId>protobuf-maven-plugin</artifactId>
                <version>0.5.1</version>
            </plugin>

        </plugins>
    </pluginManagement>

    <plugins>
        <plugin>
            <groupId>org.xolstice.maven.plugins</groupId>
            <artifactId>protobuf-maven-plugin</artifactId>
            <configuration>
                <protocArtifact>com.google.protobuf:protoc:${protobufVersion}:exe:${os.detected.classifier}</protocArtifact>
                <pluginId>grpc-java</pluginId>
                <pluginArtifact>io.grpc:protoc-gen-grpc-java:${grpcVersion}:exe:${os.detected.classifier}</pluginArtifact>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>compile</goal>
                        <goal>compile-custom</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.8.1</version>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>

    </plugins>
</build>

[sergei-ivanov]

You can safely remove maven-compiler-plugin from the build/plugins section of the parent POM. It will be ignored for packaging=pom anyway.

Try also adding this to the configuration section of protobuf-maven-plugin:

<clearOutputDirectory>false</clearOutputDirectory>

Generally, with your config, mvn clean compile on the top level module should generate protobuf messages and stubs for all configured subprojects.

[sweetodev]

Here's how my submodule POM looks like (build/plugins section) after the update, I executed mvn clean compile, not seeing generated stubs anywhere.

    <plugins>
        <plugin>
            <groupId>org.xolstice.maven.plugins</groupId>
            <artifactId>protobuf-maven-plugin</artifactId>
            <configuration>
                <protocArtifact>com.google.protobuf:protoc:${protobufVersion}:exe:${os.detected.classifier}</protocArtifact>
                <pluginId>grpc-java</pluginId>
                <pluginArtifact>io.grpc:protoc-gen-grpc-java:${grpcVersion}:exe:${os.detected.classifier}</pluginArtifact>
                <clearOutputDirectory>false</clearOutputDirectory>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>compile</goal>
                        <goal>compile-custom</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>

My ModuleA/src/main/resources/moduleA.proto looks like this

syntax = "proto3";

option java_package = "com.example.grpc";

service User {
    rpc login(Request) returns (Response);
    rpc logout(Empty) returns (Response);
}

message Request {
    string username = 1;
    string password = 2;
}

message Response {
    int32 code = 1;
    string message = 2;
}

message Empty {}

[sweetodev]

Edit just noticed this in the debug info

[INFO] --- protobuf-maven-plugin:0.5.1:compile (default) @ ModuleA ---
[INFO] /Users/username/MyProject/ModuleA/src/main/proto does not exist. Review the configuration or consider disabling the plugin.

[sergei-ivanov]

...which gives you a hint. If you place your *.proto files into src/main/proto, everything should work fine.

[sweetodev]

thanks. it works now, quick question if you don't mind, what is the right architecture to share a proto file between two separate modules (or microservices or jars), both are to be deployed on different machines, maybe compiling both modules on the same machine or perhaps compiling client on a different machine after server is deployed, what is the industry practice to share the IDL between the two, do I copy the proto file across modules and re-generate java code, just trying to wrap my head around in real life team work scenarios.

[sergei-ivanov]

I would suggest creating a shared Maven project or module that contains protobuf definitions. Protobuf compiler (and plugins such as gRPC) will generates java sources from those definitions. Those sources will then be compiled into classfiles and packaged in a jar, as normal. Protobuf definitions are also included in the jar by default (in case you need them in another project or module). Then you will have other projects or modules that will simply consume that shared module as a dependency.

[sweetodev]

Thanks Sergei, I appreciate your help.