GRPC example on Fedora 39 doesn't compile

tmulle commented 2 weeks ago

Describe the bug

When creating a new app using the quarkus start webpage which just contains GRPC extension and trying to compile it on Fedora 39 fails:

Failed to bootstrap application in NORMAL mode: Failed to resolve artifact io.grpc:protoc-gen-grpc-java:exe:linux-s390_64:1.65.1: The following artifacts could not be resolved: io.grpc:protoc-gen-grpc-java:exe:linux-s390_64:1.65.1 (absent)

We have a corporate Nexus server which is mirroring Maven Central and it should be pulling automatically, but that's another issue. The real issue is why is GRPC trying to pull unecessary dependencies?

Why is GRPC trying to pull IBM s390 on a Linux Fedora 39 system?

This is the POM.xml that was generated by the quarkus webpage:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>org.acme</groupId>
    <artifactId>code-with-quarkus</artifactId>
    <version>1.0.0-SNAPSHOT</version>

    <properties>
        <compiler-plugin.version>3.13.0</compiler-plugin.version>
        <maven.compiler.release>21</maven.compiler.release>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <quarkus.platform.artifact-id>quarkus-bom</quarkus.platform.artifact-id>
        <quarkus.platform.group-id>io.quarkus.platform</quarkus.platform.group-id>
        <quarkus.platform.version>3.13.3</quarkus.platform.version>
        <skipITs>true</skipITs>
        <surefire-plugin.version>3.2.5</surefire-plugin.version>
    </properties>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>${quarkus.platform.group-id}</groupId>
                <artifactId>${quarkus.platform.artifact-id}</artifactId>
                <version>${quarkus.platform.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <dependencies>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-grpc</artifactId>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-arc</artifactId>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-junit5</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>${quarkus.platform.group-id}</groupId>
                <artifactId>quarkus-maven-plugin</artifactId>
                <version>${quarkus.platform.version}</version>
                <extensions>true</extensions>
                <executions>
                    <execution>
                        <goals>
                            <goal>build</goal>
                            <goal>generate-code</goal>
                            <goal>generate-code-tests</goal>
                            <goal>native-image-agent</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>${compiler-plugin.version}</version>
                <configuration>
                    <parameters>true</parameters>
                </configuration>
            </plugin>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>${surefire-plugin.version}</version>
                <configuration>
                    <systemPropertyVariables>
                        <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
                        <maven.home>${maven.home}</maven.home>
                    </systemPropertyVariables>
                </configuration>
            </plugin>
            <plugin>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>${surefire-plugin.version}</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <systemPropertyVariables>
                        <native.image.path>${project.build.directory}/${project.build.finalName}-runner</native.image.path>
                        <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
                        <maven.home>${maven.home}</maven.home>
                    </systemPropertyVariables>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <profiles>
        <profile>
            <id>native</id>
            <activation>
                <property>
                    <name>native</name>
                </property>
            </activation>
            <properties>
                <skipITs>false</skipITs>
                <quarkus.native.enabled>true</quarkus.native.enabled>
            </properties>
        </profile>
    </profiles>
</project>

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

No response

Output of `uname -a` or `ver`

Linux 6.10.6-100.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Aug 19 14:35:32 UTC 2024 x86_64 GNU/Linux

Output of `java -version`

JDK 21 and JDK17

Quarkus version or git rev

3.14.1

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Maven

Additional information

No response

quarkus-bot[bot] commented 2 weeks ago

/cc @alesj (grpc), @cescoffier (grpc)

cescoffier commented 2 weeks ago

Do you get this only on Fedora 39?

We pull all os/architectures in our dependencies. We know that it's sub-optimal, but we have to do it to make sure we can build the proto files.

tmulle commented 2 weeks ago

Yes, it works fine on MacOS.

Interestingly enough, ALL dependencies get pulled no matter what the OS.

This is on my repo on MacOS which ironically gets pulled through our corporate Nexus mirror. But I don’t see the dependencies actually stored on our Nexus, but somehow it works for MacOS but fails on Fedora 39.

ls ~/.m2/repository/io/grpc/protoc-gen-grpc-java/1.65.1/
_remote.repositories                                       protoc-gen-grpc-java-1.65.1-linux-x86_64.exe.sha1
protoc-gen-grpc-java-1.65.1-linux-aarch_64.exe             protoc-gen-grpc-java-1.65.1-osx-aarch_64.exe
protoc-gen-grpc-java-1.65.1-linux-aarch_64.exe.sha1        protoc-gen-grpc-java-1.65.1-osx-aarch_64.exe.sha1
protoc-gen-grpc-java-1.65.1-linux-ppcle_64.exe             protoc-gen-grpc-java-1.65.1-osx-x86_64.exe
protoc-gen-grpc-java-1.65.1-linux-ppcle_64.exe.sha1        protoc-gen-grpc-java-1.65.1-osx-x86_64.exe.sha1
protoc-gen-grpc-java-1.65.1-linux-s390_64.exe              protoc-gen-grpc-java-1.65.1-windows-x86_32.exe
protoc-gen-grpc-java-1.65.1-linux-s390_64.exe.lastUpdated  protoc-gen-grpc-java-1.65.1-windows-x86_32.exe.sha1
protoc-gen-grpc-java-1.65.1-linux-s390_64.exe.sha1         protoc-gen-grpc-java-1.65.1-windows-x86_64.exe
protoc-gen-grpc-java-1.65.1-linux-x86_32.exe               protoc-gen-grpc-java-1.65.1-windows-x86_64.exe.sha1
protoc-gen-grpc-java-1.65.1-linux-x86_32.exe.sha1          protoc-gen-grpc-java-1.65.1.pom
protoc-gen-grpc-java-1.65.1-linux-x86_64.exe               protoc-gen-grpc-java-1.65.1.pom.sha1

cescoffier commented 2 weeks ago

I will try to get a fedora 39 to test it, but I don't have a specific nexus server.

Do you know if the issue occurs if we use a fedora 39 base docker image?

gsmet commented 1 week ago

So the problem is this thing: https://github.com/quarkusio/quarkus/blob/main/extensions/grpc/codegen/pom.xml#L28-L135 .

I think we should try to reduce the dependencies by having profiles enabled with arch.

Using os family + arch profile, I think we should be able to reduce this significantly. I don't think we need to catch all the corner cases, we could at least greatly reduce the dependencies if matching os and arch.

@tmulle maybe that's something you could create a PR for?

tmulle commented 1 week ago

@gsmet sure I can submit one. I’ll modify the pom.xml to have different profiles based on the os and arch.

cescoffier commented 1 week ago

@gsmet why did you re-open it?

cescoffier commented 1 week ago

Ok, seems to break tests...

cescoffier commented 1 week ago

I had a memory flash this morning - this is precisely why it was not done like this 3/4 years ago. I remember Michal complaining about it :-)

tmulle commented 1 week ago

That's a shame..I still get my first PR to Quarkus badge right? ;-)

So, since I'm new to the Quarkus build process why did this ultimately fail?

Did I not run the right test by using mvn -Dquickly? I guess github actions or your CI process runs deeper tests that this caught?

Why do we use Gradle and Maven together?

I did just read about Gradle and Maven profile activation and people seem to try to get around it by writing properties to files or changing the gradle builds to read the System env variables directly using if/then code branches.

Like I said in my commit/PR message I did try to read the os.detected.classifier directly in the two protoc dependencies but that didn't work because it couldn't find the variables..which is interesting because at the beginning of the builds all of those variables are printed out.

I thought you might be using this plugin which sets the variables and can be used in the builds: https://github.com/trustin/os-maven-plugin

But I'm not sure if Gradle would be able to read them..

tmulle commented 1 week ago

I just saw there is a Gradle version of the plugin I mentioned and it actually uses the same maven plugin under the hood. https://github.com/google/osdetector-gradle-plugin

So, I wonder if reading the os.detected.classifer could ultimately work.

Then no more need for profiles.

quarkusio / quarkus