search

cdsap / InfoKotlinProcess

Include Kotlin process information in Build Scans or build output
MIT License
37 stars 2 forks source link

Not output from information on GE or locally #12

Closed zoey-juan closed 1 year ago

zoey-juan commented 1 year ago

Hi, We're using the plugging but no outputs are shown. We assume It might be error occurring during the build without printing the error. https://github.com/cdsap/InfoKotlinProcess/blob/main/src/main/kotlin/io/github/cdsap/kotlinprocess/CommandLineWithOutputValue.kt#L34

cdsap commented 1 year ago

hi @zoey-juan thanks for the report. Does it happen in all builds?

rolgalan commented 1 year ago

I have the same issue when using ParallelGC. @zoey-juan is this the case for you as well?

I believe the parser is probably optimized for G1 and not considering the output for ParallelGC, which might be different. ParallelGC is still the recommended GC for Android builds, even for newer jvms.

I am using jvm 17 and this is the results based on my gradle.properties values (for reading these results, remember that G1 is default GC in jvm17 and kotlin daemon inherits gradle jvmargs when nothing is defined specifically):

I get a custom values for kotlin memory in the scan here:

org.gradle.jvmargs=-Xmx2048m -Dfile.encoding=UTF-8

I get a custom values for kotlin memory in the scan here:

org.gradle.jvmargs=-Xmx2048m -Dfile.encoding=UTF-8
kotlin.daemon.jvmargs=-Xmx1g

I don't get kotlin memory info here

org.gradle.jvmargs=-Xmx2048m -Dfile.encoding=UTF-8
kotlin.daemon.jvmargs=-Xmx1g -XX:+UseParallelGC

I don't get kotlin memory info here

org.gradle.jvmargs=-Xmx2048m -XX:+UseParallelGC -Dfile.encoding=UTF-8

I don't get kotlin memory info here

org.gradle.jvmargs=-Xmx2048m -XX:+UseParallelGC -Dfile.encoding=UTF-8
kotlin.daemon.jvmargs=-Xmx1g -XX:+UseParallelGC
cdsap commented 1 year ago

thanks @zoey-juan @rolgalan for the examples checking now

cdsap commented 1 year ago

@zoey-juan can you test with the new version 0.1.3?

@rolgalan thanks to your analysis I noticed when UseParallelGC is set, the output of jstat $process doesn't report the metrics CGC and CGCT (concurrent gc times/total).

Updated the parsing logic and released a new version, can you try 0.1.3?

rolgalan commented 1 year ago

That was fast @cdsap! I've just validated that it properly submit the info in the scans when using the ParallelGC. Thanks for for fixing this. :)

cdsap commented 1 year ago

closing this issue

zoey-juan commented 1 year ago

@cdsap sorry for late, I have tested in our project and it's still not working. Not sure how to debug since there is no logs. Any ideas? (we don't use UseParallelGC btw)

cdsap commented 1 year ago

@zoey-juan can you provide Gradle and Gradle Enterprise plugin versions?

zoey-juan commented 1 year ago

Sure, here you are Gradle: 7.6 com.gradle.enterprise: 3.10.1

cdsap commented 1 year ago

hi @zoey-juan tested with the same versions and is publishing the info: https://ge.solutions-team.gradle.com/s/5dkwajzmckdy4/custom-values#L5

I'm thinking that maybe could be a specific configuration for the JVM args. Latest version covers the confs mentioned here https://github.com/cdsap/InfoKotlinProcess/issues/12#issuecomment-1471638128

What about the Java version?

zoey-juan commented 1 year ago

hi @cdsap, kotlinOptions.jvmTarget = "1.8" and also the following parameters we're using during CI build

JAVA_OPTS: -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8
GRADLE_OPTS: >
          -Dorg.gradle.workers.max=${{ inputs.workers }}
          -Dorg.gradle.jvmargs="-Xmx${{ inputs.gradle_heap }} -XX:MaxMetaspaceSize=${{ inputs.max_metaspace_size }} -XX:+HeapDumpOnOutOfMemoryError"
          -Dkotlin.daemon.jvm.options="-Xmx${{ inputs.kotlin_heap }} -XX:MaxMetaspaceSize=${{ inputs.max_metaspace_size }} -XX:+HeapDumpOnOutOfMemoryError"
          -Dfile.encoding=utf-8
cdsap commented 1 year ago

Hi @zoey-juan I tested a different approach using your snippet in a Github action:

            - name: Execute Gradle build
              run:  ./gradlew test --tests io.github.cdsap.kotlinprocess.InfoKotlinProcessPluginTest
              env:
                GRADLE_OPTS: |
                  -Dorg.gradle.jvmargs="-Xmx1G"
                  -Dkotlin.daemon.jvm.options="-Xmx1G"

And the output of the unit test execution was:

┌─────────────────────────────────────────────────────────────────────────────┐
│  Kotlin processes                                                           │
├────────┬──────────┬───────────┬────────────┬───────────────┬────────────────┤
│  PID   │  Max     │  Usage    │  Capacity  │  GC Time      │  Uptime        │
├────────┼──────────┼───────────┼────────────┼───────────────┼────────────────┤
│  4245  │  0.5 Gb  │  0.14 Gb  │  0.22 Gb   │  0.0 minutes  │  1.45 minutes  │
├────────┼──────────┼───────────┼────────────┼───────────────┼────────────────┤
│  3366  │  1.0 Gb  │  0.1 Gb   │  0.25 Gb   │  0.0 minutes  │  2.19 minutes  │
└────────┴──────────┴───────────┴────────────┴───────────────┴────────────────┘

Notice the process 3366 allocates 1Gb of memory as the configuration is defined with GRADLE_OPTS. When I remove the GRADLE_OPTS from the configuration the output is:

┌─────────────────────────────────────────────────────────────────────────────┐
│  Kotlin processes                                                           │
├────────┬──────────┬───────────┬────────────┬───────────────┬────────────────┤
│  PID   │  Max     │  Usage    │  Capacity  │  GC Time      │  Uptime        │
├────────┼──────────┼───────────┼────────────┼───────────────┼────────────────┤
│  3943  │  0.5 Gb  │  0.06 Gb  │  0.22 Gb   │  0.0 minutes  │  1.83 minutes  │
├────────┼──────────┼───────────┼────────────┼───────────────┼────────────────┤
│  2927  │  0.5 Gb  │  0.14 Gb  │  0.22 Gb   │  0.0 minutes  │  2.67 minutes  │
└────────┴──────────┴───────────┴────────────┴───────────────┴────────────────┘

You have the results of the outputs:

In both cases, the output is shown.

The plugin retrieves the information using the JDK commands jps, jstat and jinfo, so the next steps is to verify the environment where the build is executed. Can you confirm the JDK version and vendor you're using?

Thanks!

zoey-juan commented 1 year ago

Thanks, @cdsap, We're mainly using org.jetbrains.kotlin:kotlin-stdlib-jdk7:_ but for some modules jdk8 is using. This is environment information I took from build scan.

Operating system | Linux 5.10.161+ (amd64) |  
-- | -- | --
CPU cores | 32 cores |  
Max Gradle workers | 8 workers |  
Java runtime | Debian OpenJDK Runtime Environment 11.0.18+10-post-Debian-1deb11u1 |  
Java VM | Debian OpenJDK 64-Bit Server VM 11.0.18+10-post-Debian-1deb11u1 (mixed mode, sharing) |  
Max JVM memory heap size | 12 GiB |  
Locale | English (United States) |  
Default charset | UTF-8
cdsap commented 1 year ago

thanks! I created a container using debian and jdk:

Openjdk version "11.0.18" 2023-01-17
OpenJDK Runtime Environment (build 11.0.18+10-post-Debian-1deb11u1)
OpenJDK 64-Bit Server VM (build 11.0.18+10-post-Debian-1deb11u1, mixed mode)

And the test output showed the report.

Once the environment is verified, next would be to test the plugin with a small build that involve the Kotlin compiler work. Do you have in mind a Kotlin task of one inexpensive module? Something like: ./gradlew clean :small_module:compileKotlin --no-build-cache And then check again the Build Scan > Custom values

cdsap commented 1 year ago

sorry for the delay @zoey-juan I've published a Snapshot with some logs. Instructions: 1- At the root build gradle file declare the Snapshot repository and the dependency:

buildscript {
    repositories {
        ...
        maven ( url = uri("https://s01.oss.sonatype.org/content/repositories/snapshots/") )
    }
    dependencies{
        ...
        classpath("io.github.cdsap:infokotlinprocess:0.1.4-SNAPSHOT")
    }
}

2- Apply the plugin 

apply(plugin = "io.github.cdsap.kotlinprocess")

The snapshot prints the outputs for the ValueSource and the result operation:

BUILD FAILED in 15s
158 actionable tasks: 136 executed, 22 up-to-date
JinfoResult:
-XX:CICompilerCount=4 -XX:CompressedClassSpaceSize=1065353216 -XX:ConcGCThreads=2 -XX:G1ConcRefinementThreads=9 -XX:G1HeapRegionSize=1048576 -XX:GCDrainStackTargetSize=64 -XX:InitialHeapSize=1073741824 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=6442450944 -XX:MaxMetaspaceSize=1073741824 -XX:MaxNewSize=3865051136 -XX:MinHeapDeltaBytes=1048576 -XX:NonNMethodCodeHeapSize=5836492 -XX:NonProfiledCodeHeapSize=122910874 -XX:ProfiledCodeHeapSize=122910874 -XX:ReservedCodeCacheSize=251658240 -XX:+SegmentedCodeCache -XX:-UseAOT -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC 
7979
-XX:CICompilerCount=4 -XX:CompressedClassSpaceSize=1065353216 -XX:ConcGCThreads=2 -XX:G1ConcRefinementThreads=9 -XX:G1HeapRegionSize=1048576 -XX:GCDrainStackTargetSize=64 -XX:InitialHeapSize=1073741824 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=6442450944 -XX:MaxMetaspaceSize=1073741824 -XX:MaxNewSize=3865051136 -XX:MinHeapDeltaBytes=1048576 -XX:NonNMethodCodeHeapSize=5836492 -XX:NonProfiledCodeHeapSize=122910874 -XX:ProfiledCodeHeapSize=122910874 -XX:ReservedCodeCacheSize=251658240 -XX:+SegmentedCodeCache -XX:-UseAOT -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC 
59350

jStatResult:
Timestamp        S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT    CGC    CGCT     GCT   
          569.6  0.0   18432.0  0.0   18432.0 187392.0 34816.0   842752.0   57545.5   77872.0 76573.5 9088.0 8547.5      9    0.055   0      0.000   6      0.007    0.062
7979
Timestamp        S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT    CGC    CGCT     GCT   
        24234.1  0.0   45056.0  0.0   45056.0 333824.0 191488.0  510976.0   354104.6  956464.0 811051.3 127488.0 99754.8    239    9.934   3      1.957  108     7.290   19.181
59350

Publishing build scan...
https://xxxx.com/s/b2552jwmyoukw
zoey-juan commented 1 year ago

Hi @cdsap , Sorry for late. I have tested the patch and I checked all build scans it's working locally but not working on our CI environment. (All build scans from local builds have the custom values but not the build scans from CI.)

This is the log result from our CI.

JinfoResult:

jStatResult:

Publishing build scan...
[https://ge.xxx.xxx]

Information from the build scan Gradle 7.6, Gradle Enterprise plugin 3.10.1, Android Gradle plugin 7.4.0 Composite build (1 included build)

Infrastructure

Operating system      Linux 5.10.162+ (amd64)   
CPU cores                  32 cores 
Max Gradle workers  8 workers   
Java runtime              Debian OpenJDK Runtime Environment 11.0.18+10-post-Debian-1deb11u1    
Java VM                    Debian OpenJDK 64-Bit Server VM 11.0.18+10-post-Debian-1deb11u1 (mixed mode, sharing)    
Max JVM memory heap size    12 GiB  
Locale                     English (United States)  
Default charset   UTF-8 
Username              runner

Custom Values

CI run  492xxxxx
CI workflow On pull request 
Git branch  HEAD    
Git commit id   xxxx
Git commit id short xxxx    
Git repository  https://github.com/xxxx
Git status  ?? keys/debugkey.gradle
?? keys/stgkey.gradle
?? keys/testkey.gradle
?? secret-manager.json  
os.arch amd64   
os.name Linux

Any ideas of the reasons?

zoey-juan commented 1 year ago

I found this is probably the main reason it's probably we did not config properly. The data starts displaying after we change the config from

   GRADLE_OPTS: >
          -Dorg.gradle.workers.max=${{ inputs.workers }}
          -Dorg.gradle.jvmargs="-Xmx${{ inputs.gradle_heap }} -XX:MaxMetaspaceSize=${{ inputs.max_metaspace_size }} -XX:+HeapDumpOnOutOfMemoryError"
          -Dkotlin.daemon.jvm.options="-Xmx${{ inputs.kotlin_heap }} -XX:MaxMetaspaceSize=${{ inputs.max_metaspace_size }} -XX:+HeapDumpOnOutOfMemoryError"
          -Dfile.encoding=utf-8

to (by refer to kotlin doc)

   GRADLE_OPTS: >
          -Dorg.gradle.workers.max=${{ inputs.workers }}
          -Dorg.gradle.jvmargs="-Xmx${{ inputs.gradle_heap }} -XX:MaxMetaspaceSize=${{ inputs.max_metaspace_size }} -XX:+HeapDumpOnOutOfMemoryError -Dkotlin.daemon.jvm.options=-Xmx${{ inputs.kotlin_heap }}"
      -Dfile.encoding=utf-8

I'm closing this issue, thank you @cdsap !

-- FYI, the original config we had above is also causing a long build error log from kotlin daemon errors.

Could not connect to kotlin daemon. Using fallback strategy.
e: Daemon compilation failed: Could not connect to Kotlin compile daemon
java.lang.RuntimeException: Could not connect to Kotlin compile daemon
    at org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork.compileWithDaemon(GradleKotlinCompilerWork.kt:204)
    at org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork.compileWithDaemonOrFallbackImpl(GradleKotlinCompilerWork.kt:161)
    at org.jetbrains.kotlin.compilerRunner.GradleKotlinCompilerWork.run(GradleKotlinCompilerWork.kt:132)