search

knowledgesystems / pipelines-scrum

Repository for tracking uncategorizable issues related to backend pipelines work
0 stars 0 forks source link

Migrate onto Java21 #1230

Closed averyniceday closed 8 months ago

averyniceday commented 8 months ago

Done Condition (What do we need? Why do we need it? Keep this is small as possible!)

Technical Description (How are we going to achieve the above)

Potential Issues

Dependencies

Technical Requirements

Outside People/Teams

Changes

mandawilson commented 8 months ago

PRs (but update them with the latest cbioportal before merging them):

mandawilson commented 8 months ago

CMO pipelines (though some of these applied to pipelines and/or genome-nexus-annotation-pipeline):

Upgrade to Spring 5.3.31image.png : Spring JDK version support https://github.com/spring-projects/spring-framework/wiki/Spring-Framework-Versions# Spring Framework 5.3.x: JDK 8-21 (as of 5.3.26)

Upgrade to Spring boot 2.7.18: Looks like spring boot is 2.3.3.RELEASE, but that according to this: https://spring.io/blog/2023/11/16/spring-framework-5-3-31-and-6-0-14-available-now/ Spring Framework 5.3.31 comes with Spring Boot 2.7.18

Switch to mysql.com mysql-connector-j Only needed for cmo-pipelines. Pipelines didn't need this update but we did it anyway: [ERROR] 'dependencies.dependency.version' for mysql:mysql-connector-java:jar is missing. @ line 51, column 17

Update to 3.5.1 for the maven shade plugin: Unable to parse configuration of mojo org.apache.maven.plugins:maven-shade-plugin:2.3:shade for parameter transformers: Cannot load implementation hint 'org.springframework.boot.maven.PropertiesMergingResourceTransformer': org/apache/maven/plugins/shade/resource/ReproducibleResourceTransformer: org.apache.maven.plugins.shade.resource.ReproducibleResourceTransformer

This says to update to latest version of plugin: https://stackoverflow.com/questions/55748745/unable-to-parse-configuration-of-mojo-org-apache-maven-pluginsmaven-shade-plugi

mandawilson commented 8 months ago

genome-nexus-annotation-pipeline:

Updating byte-buddy + mockito because of this: Caused by: java.lang.IllegalArgumentException: Java 21 (65) is not supported by the current version of Byte Buddy which officially supports Java 20 (64) - update Byte Buddy or set net.bytebuddy.experimental as a VM property Solution here: https://github.com/raphw/byte-buddy/issues/1396

I had trouble using a updated genome-nexus-java-api-client because I had to use a fork of the library we use to compile to code in order to compile with Java 21, and then it was throwing exceptions in the genome-nexus-annotation-pipeline because fields defined as non-optional/required in the GN swagger were not being returned to us on API calls.

mandawilson commented 8 months ago

We were getting this error compiling: [ERROR] Failed to execute goal on project importer: Could not resolve dependencies for project org.mskcc.cbio:importer:jar:1.0-SNAPSHOT: Failed to collect dependencies at com.github.cbioportal.cbioportal:core:jar:d4b0655 -> com.github.cbioportal.cbioportal:security-spring:jar:d4b0655 -> com.github.cbioportal:spring-social-live:jar:40100de1ad80e2865c834f2b64b32bb6c2b368ce -> org.springframework.social:spring-social-security:jar:1.1.0.M4: Failed to read artifact descriptor for org.springframework.social:spring-social-security:jar:1.1.0.M4: Could not transfer artifact org.springframework.social:spring-social-security:pom:1.1.0.M4 from/to maven-default-http-blocker (http://0.0.0.0/): Blocked mirror for repositories: [spring-milestones (http://repo.spring.io/milestone, default, releases), org.springframework.maven.release (http://maven.springframework.org/release, default, releases), org.springframework.maven.milestone (http://maven.springframework.org/milestone, default, releases)] -> [Help 1]

We manually downloaded the jar and pom from here: https://repo.spring.io/ui/native/plugins-release/org/springframework/social/spring-social-security/1.1.0.M4/

mvn install:install-file -Dfile=spring-social-security-1.1.0.M4.jar -DpomFile=spring-social-security-1.1.0.M4.pom

We have done this for spring-social-security, spring-social-core, and spring-social-web.

mvn install:install-file -Dfile=spring-social-core-1.1.0.M4.jar -DpomFile=spring-social-core-1.1.0.M4.pom

We manually unpacked the jar in /home/wilson/.m2/repository/batik/batik/1.5/ and corrected the MANIFEST.MF file. We also had to for batik-rasterizer-1.5.jar.

DON'T DO A mvn -U clean install

jar xf batik-rasterizer-1.5.jar

jar cf batik-rasterizer-1.5.jar .

If there are other jar files then use 0 as an option.

sheridancbio commented 8 months ago

we moved the "deploy jars into production and update pipelines3 to a java21 jvm" type of tasks to the card for integrating pipelines with the new "cbioportal-core" repository.

sheridancbio commented 8 months ago

As things stand, both the cmo-pipelines repository and the pipelines repository have been compiled and tested (at least unit tests) under a java21 jvm. Bringing both codebases into accord with the latest version of the cbioportal codebase now requires integration with the externalized core module which is available now here: https://github.com/cBioPortal/cbioportal-core

In the current build, we are using these somewhat out of date timepoints:

cmo-pipelines

This PR https://github.com/knowledgesystems/cmo-pipelines/pull/1100 builds off of the head of the master branch of cmo-pipelines. It links in dependency genome-nexus-annotation-pipeline at commit hash 0f4e72e5902e309f082083068b17bffaf2a86565 which was merged on (2023_10_11 via pr https://github.com/genome-nexus/genome-nexus-annotation-pipeline/pull/267). That in turn has dependency on cbioportal at commit hash ee5802d836c05ed846d7d1ea3f584febdc07ffa8 which was merged on (2023_10_10). So this development is about 3 months out of currency, and requires adding the new cbioportal-core repository to be made current.

pipelines

This PR https://github.com/knowledgesystems/pipelines/pull/460 builds off of the head of the master branch of pipelines. It links in dependency genome-nexus-annotation-pipeline at commit hash efd80bbbee60826dabfab1a8a26b1e2969a90508 which is part of a pending PR to genome-nexus-annotation-pipeline https://github.com/genome-nexus/genome-nexus-annotation-pipeline/pull/272 which is built off the head of master. That in turn has dependency on cbioportal at commit hash ee5802d836c05ed846d7d1ea3f584febdc07ffa8 which was merged on (2023_10_10). A second direct dependency is present in pipelines on that same point in the history of cbioportal (ee5802d836c05ed846d7d1ea3f584febdc07ffa8) So this development is also about 3 months out of currency, and requires adding the new cbioportal-core repository to be made current.

mandawilson commented 8 months ago

The genome-nexus-annotation-pipeline also compiles and runs tests under the Java 21 jvm.