quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.83k stars 2.7k forks source link

Support protoc Kotlin code generation #44552

Open PhilKes opened 4 days ago

PhilKes commented 4 days ago

Closes #39127

This PR adds support to generate Kotlin DSLs/Extensions for protoc via adding the kotlin_out parameter to the protoc command:

To test it I reused the kotlin-grpc-project integration-test to use the generated DSL for HelloMsg. Now currently when running the KotlinGRPCProjectBuildTest, compileKotlin fails with:

e: file://.../quarkus/integration-tests/gradle/target/classes/kotlin-grpc-project/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:17 Annotation argument must be a compile-time constant.
e: file://.../quarkus/integration-tests/gradle/target/classes/kotlin-grpc-project/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:37 Unresolved reference 'kotlin'.
e: file://.../quarkus/integration-tests/gradle/target/classes/kotlin-grpc-project/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:16:24 Unresolved reference 'kotlin'.
HelloMsgKt.kt ```kotlin // Generated by the protocol buffer compiler. DO NOT EDIT! // source: hello.proto // Generated files should ignore deprecation warnings @file:Suppress("DEPRECATION") package io.quarkus.example; @kotlin.jvm.JvmName("-initializehelloMsg") public inline fun helloMsg(block: io.quarkus.example.HelloMsgKt.Dsl.() -> kotlin.Unit): io.quarkus.example.HelloMsg = io.quarkus.example.HelloMsgKt.Dsl._create(io.quarkus.example.HelloMsg.newBuilder()).apply { block() }._build() /** * Protobuf type `io.quarkus.example.HelloMsg` */ public object HelloMsgKt { @kotlin.OptIn(com.google.protobuf.kotlin.OnlyForUseByGeneratedProtoCode::class) @com.google.protobuf.kotlin.ProtoDslMarker public class Dsl private constructor( private val _builder: io.quarkus.example.HelloMsg.Builder ) { public companion object { @kotlin.jvm.JvmSynthetic @kotlin.PublishedApi internal fun _create(builder: io.quarkus.example.HelloMsg.Builder): Dsl = Dsl(builder) } @kotlin.jvm.JvmSynthetic @kotlin.PublishedApi internal fun _build(): io.quarkus.example.HelloMsg = _builder.build() /** * `string message = 1;` */ public var message: kotlin.String @JvmName("getMessage") get() = _builder.getMessage() @JvmName("setMessage") set(value) { _builder.setMessage(value) } /** * `string message = 1;` */ public fun clearMessage() { _builder.clearMessage() } /** * `.google.protobuf.Timestamp date_time = 2;` */ public var dateTime: com.google.protobuf.Timestamp @JvmName("getDateTime") get() = _builder.getDateTime() @JvmName("setDateTime") set(value) { _builder.setDateTime(value) } /** * `.google.protobuf.Timestamp date_time = 2;` */ public fun clearDateTime() { _builder.clearDateTime() } /** * `.google.protobuf.Timestamp date_time = 2;` * @return Whether the dateTime field is set. */ public fun hasDateTime(): kotlin.Boolean { return _builder.hasDateTime() } /** * `.io.quarkus.example.HelloMsg.Status status = 3;` */ public var status: io.quarkus.example.HelloMsg.Status @JvmName("getStatus") get() = _builder.getStatus() @JvmName("setStatus") set(value) { _builder.setStatus(value) } public var statusValue: kotlin.Int @JvmName("getStatusValue") get() = _builder.getStatusValue() @JvmName("setStatusValue") set(value) { _builder.setStatusValue(value) } /** * `.io.quarkus.example.HelloMsg.Status status = 3;` */ public fun clearStatus() { _builder.clearStatus() } } } @kotlin.jvm.JvmSynthetic public inline fun io.quarkus.example.HelloMsg.copy(block: `io.quarkus.example`.HelloMsgKt.Dsl.() -> kotlin.Unit): io.quarkus.example.HelloMsg = `io.quarkus.example`.HelloMsgKt.Dsl._create(this.toBuilder()).apply { block() }._build() public val io.quarkus.example.HelloMsgOrBuilder.dateTimeOrNull: com.google.protobuf.Timestamp? get() = if (hasDateTime()) getDateTime() else null ```

I am guessing this is because the generated kotlin code should not be in build/classes/java but rather build/classes/kotlin? But if we provide a different output path for the kotlin_out than for java_out we would need a way to add this extra path also to the compile sources, which is done in CodeGenerator.init(), right?

github-actions[bot] commented 4 days ago

🎊 PR Preview 3dff5a44763679bbd38e11c69ec16280593f20b0 has been successfully built and deployed to https://quarkus-pr-main-44552-preview.surge.sh/version/main/guides/

quarkus-bot[bot] commented 4 days ago

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

Status for workflow Quarkus Documentation CI

This is the status report for running Quarkus Documentation CI on commit 3227bb7e6f0eb6f3db5694bd071c76fae459a12c.

:white_check_mark: The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

[!WARNING] There are other workflow runs running, you probably need to wait for their status before merging.

quarkus-bot[bot] commented 4 days ago

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

Status for workflow Quarkus CI

This is the status report for running Quarkus CI on commit 3227bb7e6f0eb6f3db5694bd071c76fae459a12c.

Failing Jobs

Status Name Step Failures Logs Raw logs Build scan
Gradle Tests - JDK 17 Build Failures Logs Raw logs :mag:
Gradle Tests - JDK 17 Windows Build Failures Logs Raw logs :mag:

Full information is available in the Build summary check run. You can consult the Develocity build scans.

Failures

:gear: Gradle Tests - JDK 17 #

- Failing: integration-tests/gradle 

:package: integration-tests/gradle

io.quarkus.gradle.KotlinGRPCProjectBuildTest.testBasicMultiModuleBuild - History - More details - Source on GitHub

``` java.lang.AssertionError: Gradle build failed with exit code 1 at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:140) at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:57) at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:52) at app//io.quarkus.gradle.KotlinGRPCProjectBuildTest.testBasicMultiModuleBuild(KotlinGRPCProjectBuildTest.java:14) at java.base@17.0.13/java.lang.reflect.Method.invoke(Method.java:569) at java.base@17.0.13/java.util.ArrayList.forEach(ArrayList.java:1511) at java.base@17.0.13/java.util.ArrayList.forEach(ArrayList.java:1511) ```

io.quarkus.gradle.devmode.KotlinProjectWithCompilerArgsDevModeTest.main - History - More details - Source on GitHub

``` java.lang.AssertionError: Expecting actual: "BROKEN: quarkusDev mode has terminated" to contain: "[Banana!]!" at app//io.quarkus.gradle.devmode.KotlinProjectWithCompilerArgsDevModeTest.testDevMode(KotlinProjectWithCompilerArgsDevModeTest.java:17) at app//io.quarkus.gradle.devmode.QuarkusDevGradleTestBase.main(QuarkusDevGradleTestBase.java:60) at java.base@17.0.13/java.lang.reflect.Method.invoke(Method.java:569) ```

:gear: Gradle Tests - JDK 17 Windows #

- Failing: integration-tests/gradle 

:package: integration-tests/gradle

io.quarkus.gradle.KotlinGRPCProjectBuildTest.testBasicMultiModuleBuild - History - More details - Source on GitHub

``` java.lang.AssertionError: Gradle build failed with exit code 1 at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:140) at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:57) at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:52) at app//io.quarkus.gradle.KotlinGRPCProjectBuildTest.testBasicMultiModuleBuild(KotlinGRPCProjectBuildTest.java:14) at java.base@17.0.13/java.lang.reflect.Method.invoke(Method.java:569) at java.base@17.0.13/java.util.ArrayList.forEach(ArrayList.java:1511) at java.base@17.0.13/java.util.ArrayList.forEach(ArrayList.java:1511) ```

io.quarkus.gradle.devmode.KotlinProjectWithCompilerArgsDevModeTest.main - History - More details - Source on GitHub

``` java.lang.AssertionError: Expecting actual: "BROKEN: quarkusDev mode has terminated" to contain: "[Banana!]!" at app//io.quarkus.gradle.devmode.KotlinProjectWithCompilerArgsDevModeTest.testDevMode(KotlinProjectWithCompilerArgsDevModeTest.java:17) at app//io.quarkus.gradle.devmode.QuarkusDevGradleTestBase.main(QuarkusDevGradleTestBase.java:60) at java.base@17.0.13/java.lang.reflect.Method.invoke(Method.java:569) ```
gsmet commented 4 days ago

@cdsap I'm not sure if it's related to this change but I have seen the following in the logs and it doesn't look like something we would want:

2024-11-17T17:13:44.2065823Z 2 problems were found storing the configuration cache. 2024-11-17T17:13:44.2068318Z - Task :quarkusDev of type io.quarkus.gradle.tasks.QuarkusDev: cannot serialize object of type 'org.gradle.api.internal.artifacts.configurations.DefaultUnlockedConfiguration', a subtype of 'org.gradle.api.artifacts.Configuration', as these are not supported with the configuration cache. 2024-11-17T17:13:44.2069077Z See https://docs.gradle.org/8.9/userguide/configuration_cache.html#config_cache:requirements:disallowed_types 2024-11-17T17:13:44.2071428Z - Task :quarkusDev of type io.quarkus.gradle.tasks.QuarkusDev: cannot serialize object of type 'org.gradle.api.internal.tasks.DefaultSourceSet', a subtype of 'org.gradle.api.tasks.SourceSet', as these are not supported with the configuration cache. 2024-11-17T17:13:44.2072168Z See https://docs.gradle.org/8.9/userguide/configuration_cache.html#config_cache:requirements:disallowed_types

cdsap commented 3 days ago

@gsmet I tested with a version without the configuration cache changes (3.15.1), I'm observing the same log: https://ge.solutions-team.gradle.com/s/g33okmapudxeg/console-log?page=1#L73

Note that I'm executing the build like: ./gradlew clean quarkusDev --configuration-cache --no-build-cache

Will confirm with the team

PhilKes commented 2 days ago

This looks promising, thanks for the PR.

I suggested some minor doc changes and more importantly it seems some Gradle tests are failing and I think that's the important bit:

2024-11-17T17:13:44.2059374Z Options for KOTLIN DAEMON: IncrementalCompilationOptions(super=CompilationOptions(compilerMode=INCREMENTAL_COMPILER, targetPlatform=JVM, reportCategories=[0, 3], reportSeverity=2, requestedCompilationResults=[0], kotlinScriptExtensions=[]), areFileChangesKnown=false, modifiedFiles=null, deletedFiles=null, classpathChanges=NotAvailableForNonIncrementalRun, workingDir=/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/cacheable, multiModuleICSettings=MultiModuleICSettings(buildHistoryFile=/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/local-state/build-history.bin, useModuleDetection=false), usePreciseJavaTracking=true, icFeatures=IncrementalCompilationFeatures(withAbiSnapshot=false, preciseCompilationResultsBackup=true, keepIncrementalCompilationCachesInMemory=true, enableUnsafeIncrementalCompilationForMultiplatform=false), outputFiles=[/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/kotlin/main, /home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/cacheable, /home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/local-state])
2024-11-17T17:13:44.2061194Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:17 Annotation argument must be a compile-time constant.
2024-11-17T17:13:44.2063191Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:37 Unresolved reference 'kotlin'.
2024-11-17T17:13:44.2065030Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:16:24 Unresolved reference 'kotlin'.
2024-11-17T17:13:44.2065363Z Finished executing kotlin compiler using DAEMON strategy

Thanks for the feedback. I am aware of the failing tests, as I described in the PR description I am not sure what the problem is since I dont have much experienced with mixed java/kotlin generated sources. Do the generated kotlin sources have to be in another folder than the generated java code? (see my original PR description)

quarkus-bot[bot] commented 2 days ago

Status for workflow Quarkus Documentation CI

This is the status report for running Quarkus Documentation CI on commit 8c47257dd70f7b45f59096b7c6ade19716e38489.

:white_check_mark: The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

[!WARNING] There are other workflow runs running, you probably need to wait for their status before merging.

quarkus-bot[bot] commented 2 days ago

Status for workflow Quarkus CI

This is the status report for running Quarkus CI on commit 8c47257dd70f7b45f59096b7c6ade19716e38489.

Failing Jobs

Status Name Step Failures Logs Raw logs Build scan
Gradle Tests - JDK 17 Build Failures Logs Raw logs :mag:
Gradle Tests - JDK 17 Windows Build Failures Logs Raw logs :mag:

Full information is available in the Build summary check run. You can consult the Develocity build scans.

Failures

:gear: Gradle Tests - JDK 17 #

- Failing: integration-tests/gradle 

:package: integration-tests/gradle

io.quarkus.gradle.KotlinGRPCProjectBuildTest.testBasicMultiModuleBuild - History - More details - Source on GitHub

``` java.lang.AssertionError: Gradle build failed with exit code 1 at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:140) at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:57) at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:52) at app//io.quarkus.gradle.KotlinGRPCProjectBuildTest.testBasicMultiModuleBuild(KotlinGRPCProjectBuildTest.java:14) at java.base@17.0.13/java.lang.reflect.Method.invoke(Method.java:569) at java.base@17.0.13/java.util.ArrayList.forEach(ArrayList.java:1511) at java.base@17.0.13/java.util.ArrayList.forEach(ArrayList.java:1511) ```

io.quarkus.gradle.devmode.KotlinProjectWithCompilerArgsDevModeTest.main - History - More details - Source on GitHub

``` java.lang.AssertionError: Expecting actual: "BROKEN: quarkusDev mode has terminated" to contain: "[Banana!]!" at app//io.quarkus.gradle.devmode.KotlinProjectWithCompilerArgsDevModeTest.testDevMode(KotlinProjectWithCompilerArgsDevModeTest.java:17) at app//io.quarkus.gradle.devmode.QuarkusDevGradleTestBase.main(QuarkusDevGradleTestBase.java:60) at java.base@17.0.13/java.lang.reflect.Method.invoke(Method.java:569) ```

:gear: Gradle Tests - JDK 17 Windows #

- Failing: integration-tests/gradle 

:package: integration-tests/gradle

io.quarkus.gradle.KotlinGRPCProjectBuildTest.testBasicMultiModuleBuild - History - More details - Source on GitHub

``` java.lang.AssertionError: Gradle build failed with exit code 1 at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:140) at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:57) at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:52) at app//io.quarkus.gradle.KotlinGRPCProjectBuildTest.testBasicMultiModuleBuild(KotlinGRPCProjectBuildTest.java:14) at java.base@17.0.13/java.lang.reflect.Method.invoke(Method.java:569) at java.base@17.0.13/java.util.ArrayList.forEach(ArrayList.java:1511) at java.base@17.0.13/java.util.ArrayList.forEach(ArrayList.java:1511) ```

io.quarkus.gradle.devmode.KotlinProjectWithCompilerArgsDevModeTest.main - History - More details - Source on GitHub

``` java.lang.AssertionError: Expecting actual: "BROKEN: quarkusDev mode has terminated" to contain: "[Banana!]!" at app//io.quarkus.gradle.devmode.KotlinProjectWithCompilerArgsDevModeTest.testDevMode(KotlinProjectWithCompilerArgsDevModeTest.java:17) at app//io.quarkus.gradle.devmode.QuarkusDevGradleTestBase.main(QuarkusDevGradleTestBase.java:60) at java.base@17.0.13/java.lang.reflect.Method.invoke(Method.java:569) ```