Dependency updates - Githubissues

dgault commented 2 years ago

Following on from conversation with @sbesson and @jburel, trying to remove some of the dependencies. Goal of this PR will be to update the below areas:

1 - formats API bumped to latest 6.10.0, this should be straight forward 2 - commons-io removed, only area impacted will be the list of used files. PR aims to keep the reproduce the same list 3 - initially remove netty-nio-client dependency, this looks safe and straight forward

As a follow up I will continue trying to strip out and remove the remaining amazon dependencies which essentially means removing the s3 functionality for the time being.

jburel commented 2 years ago

Building this PR in https://github.com/ome/openmicroscopy/pull/6324 (I used my branch since this branch is not built on top of main and the version is old). It now fails with new error

        ::::::::::::::::::::::::::::::::::::::::::::::
        ::              FAILED DOWNLOADS            ::
        :: ^ see resolution messages for details  ^ ::
        ::::::::::::::::::::::::::::::::::::::::::::::
        :: org.objenesis#objenesis;2.5.1!objenesis.jar
        :: joda-time#joda-time;2.2!joda-time.jar
        :: com.google.guava#guava;29.0-jre!guava.jar(bundle)
        :: org.checkerframework#checker-qual;2.11.1!checker-qual.jar
        :: com.google.errorprone#error_prone_annotations;2.3.4!error_prone_annotations.jar
        :: ch.qos.logback#logback-core;1.2.0!logback-core.jar
        ::::::::::::::::::::::::::::::::::::::::::::::

sbesson commented 2 years ago

Have you established that removing/commenting https://github.com/ome/openmicroscopy/blob/f73008bc34d37f6034a21095c55eb2a8b6075e44/ivy.xml#L21 suffices to fix the build issues? i.e. is this component the only source of issues?

jburel commented 2 years ago

I have rebuilt using only https://github.com/ome/openmicroscopy/pull/6323 and excluding the ZarrReader and it builds fine

dgault commented 2 years ago

Heres the dependency tree with this PR, other than logback the rest of these are related to jzarr:

[INFO] ome:OMEZarrReader:jar:0.1.6-SNAPSHOT
[INFO] +- com.bc.zarr:jzarr:jar:0.3.4:compile
[INFO] |  +- edu.ucar:cdm-core:jar:5.3.3:compile
[INFO] |  |  +- edu.ucar:udunits:jar:5.3.3:compile
[INFO] |  |  +- edu.ucar:httpservices:jar:5.3.3:compile
[INFO] |  |  |  +- org.apache.httpcomponents:httpclient:jar:4.5.9:compile
[INFO] |  |  |  |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  |  |  |  \- commons-codec:commons-codec:jar:1.11:compile
[INFO] |  |  |  \- org.apache.httpcomponents:httpmime:jar:4.5.9:compile
[INFO] |  |  +- joda-time:joda-time:jar:2.10.3:compile
[INFO] |  |  +- org.jdom:jdom2:jar:2.0.6:compile
[INFO] |  |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  |  +- com.google.protobuf:protobuf-java:jar:3.9.1:compile
[INFO] |  |  +- com.google.guava:guava:jar:28.0-jre:compile
[INFO] |  |  |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  |  |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  |  |  +- org.checkerframework:checker-qual:jar:2.8.1:compile
[INFO] |  |  |  +- com.google.errorprone:error_prone_annotations:jar:2.3.2:compile
[INFO] |  |  |  +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] |  |  |  \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
[INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.11:compile
[INFO] |  |  \- com.google.re2j:re2j:jar:1.3:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.11.3:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.11.3:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.11.3:compile
[INFO] |  \- org.lasersonlab:jblosc:jar:1.0.1:compile
[INFO] |     \- net.java.dev.jna:jna:jar:4.2.2:compile
[INFO] +- org.mockito:mockito-inline:jar:3.7.7:compile
[INFO] |  \- org.mockito:mockito-core:jar:3.7.7:compile
[INFO] |     +- net.bytebuddy:byte-buddy:jar:1.10.19:compile
[INFO] |     +- net.bytebuddy:byte-buddy-agent:jar:1.10.19:compile
[INFO] |     \- org.objenesis:objenesis:jar:3.1:compile
[INFO] +- org.testng:testng:jar:6.8:compile
[INFO] |  +- junit:junit:jar:4.10:compile
[INFO] |  |  \- org.hamcrest:hamcrest-core:jar:1.1:compile
[INFO] |  +- org.beanshell:bsh:jar:2.0b4:compile
[INFO] |  +- com.beust:jcommander:jar:1.27:compile
[INFO] |  \- org.yaml:snakeyaml:jar:1.6:compile
[INFO] +- xalan:serializer:jar:2.7.2:runtime
[INFO] |  \- xml-apis:xml-apis:jar:1.3.04:runtime
[INFO] +- xalan:xalan:jar:2.7.2:runtime
[INFO] \- ome:formats-api:jar:6.10.0:compile
[INFO]    +- org.openmicroscopy:ome-common:jar:6.0.9:compile
[INFO]    |  +- io.minio:minio:jar:5.0.2:compile
[INFO]    |  |  +- com.google.http-client:google-http-client-xml:jar:1.20.0:compile
[INFO]    |  |  |  +- com.google.http-client:google-http-client:jar:1.20.0:compile
[INFO]    |  |  |  \- xpp3:xpp3:jar:1.1.4c:compile
[INFO]    |  |  +- com.squareup.okhttp3:okhttp:jar:3.7.0:compile
[INFO]    |  |  \- com.squareup.okio:okio:jar:1.12.0:compile
[INFO]    |  +- com.esotericsoftware:kryo:jar:4.0.2:compile
[INFO]    |  |  +- com.esotericsoftware:reflectasm:jar:1.11.3:compile
[INFO]    |  |  |  \- org.ow2.asm:asm:jar:5.0.4:compile
[INFO]    |  |  \- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO]    |  +- ch.qos.logback:logback-core:jar:1.2.9:compile
[INFO]    |  \- ch.qos.logback:logback-classic:jar:1.2.0:compile
[INFO]    +- org.openmicroscopy:ome-xml:jar:6.3.1:compile
[INFO]    |  \- org.openmicroscopy:specification:jar:6.3.1:compile
[INFO]    +- org.openmicroscopy:ome-codecs:jar:0.3.2:compile
[INFO]    |  \- org.openmicroscopy:ome-jai:jar:0.1.3:compile
[INFO]    \- org.slf4j:slf4j-api:jar:1.7.6:compile

sbesson commented 2 years ago

Using % mvn dependency:tree -Dverbose=true to include the conflicting dependency (resolved by the build system)

[INFO] ome:OMEZarrReader:jar:0.2.1-SNAPSHOT
[INFO] +- com.bc.zarr:jzarr:jar:0.3.4:compile
[INFO] |  +- edu.ucar:cdm-core:jar:5.3.3:compile
[INFO] |  |  +- edu.ucar:udunits:jar:5.3.3:compile
[INFO] |  |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.2:compile - omitted for duplicate)
[INFO] |  |  |  \- (org.slf4j:slf4j-api:jar:1.7.28:compile - omitted for duplicate)
[INFO] |  |  +- edu.ucar:httpservices:jar:5.3.3:compile
[INFO] |  |  |  +- (com.google.guava:guava:jar:28.0-jre:compile - omitted for duplicate)
[INFO] |  |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.2:compile - omitted for duplicate)
[INFO] |  |  |  +- org.apache.httpcomponents:httpclient:jar:4.5.9:compile
[INFO] |  |  |  |  +- (org.apache.httpcomponents:httpcore:jar:4.4.11:compile - omitted for duplicate)
[INFO] |  |  |  |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  |  |  |  \- commons-codec:commons-codec:jar:1.11:compile
[INFO] |  |  |  +- (org.apache.httpcomponents:httpcore:jar:4.4.11:compile - omitted for duplicate)
[INFO] |  |  |  +- org.apache.httpcomponents:httpmime:jar:4.5.9:compile
[INFO] |  |  |  |  \- (org.apache.httpcomponents:httpclient:jar:4.5.9:compile - omitted for duplicate)
[INFO] |  |  |  +- (com.google.re2j:re2j:jar:1.3:compile - omitted for duplicate)
[INFO] |  |  |  \- (org.slf4j:slf4j-api:jar:1.7.28:compile - omitted for duplicate)
[INFO] |  |  +- joda-time:joda-time:jar:2.10.3:compile
[INFO] |  |  +- org.jdom:jdom2:jar:2.0.6:compile
[INFO] |  |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  |  +- com.google.protobuf:protobuf-java:jar:3.9.1:compile
[INFO] |  |  +- com.google.guava:guava:jar:28.0-jre:compile
[INFO] |  |  |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  |  |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  |  |  +- (com.google.code.findbugs:jsr305:jar:3.0.2:compile - omitted for duplicate)
[INFO] |  |  |  +- org.checkerframework:checker-qual:jar:2.8.1:compile
[INFO] |  |  |  +- com.google.errorprone:error_prone_annotations:jar:2.3.2:compile
[INFO] |  |  |  +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] |  |  |  \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
[INFO] |  |  +- (com.beust:jcommander:jar:1.78:compile - omitted for conflict with 1.27)
[INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.11:compile
[INFO] |  |  +- com.google.re2j:re2j:jar:1.3:compile
[INFO] |  |  \- (org.slf4j:slf4j-api:jar:1.7.28:compile - omitted for conflict with 1.7.6)
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.11.3:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.11.3:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.11.3:compile
[INFO] |  \- org.lasersonlab:jblosc:jar:1.0.1:compile
[INFO] |     +- (junit:junit:jar:4.12:compile - omitted for conflict with 4.10)
[INFO] |     \- net.java.dev.jna:jna:jar:4.2.2:compile
[INFO] +- org.mockito:mockito-inline:jar:3.7.7:compile
[INFO] |  \- org.mockito:mockito-core:jar:3.7.7:compile
[INFO] |     +- net.bytebuddy:byte-buddy:jar:1.10.19:compile
[INFO] |     +- net.bytebuddy:byte-buddy-agent:jar:1.10.19:compile
[INFO] |     \- org.objenesis:objenesis:jar:3.1:compile
[INFO] +- org.testng:testng:jar:6.8:compile
[INFO] |  +- junit:junit:jar:4.10:compile
[INFO] |  |  \- org.hamcrest:hamcrest-core:jar:1.1:compile
[INFO] |  +- org.beanshell:bsh:jar:2.0b4:compile
[INFO] |  +- com.beust:jcommander:jar:1.27:compile
[INFO] |  \- org.yaml:snakeyaml:jar:1.6:compile
[INFO] +- xalan:serializer:jar:2.7.2:runtime
[INFO] |  \- xml-apis:xml-apis:jar:1.3.04:runtime
[INFO] +- xalan:xalan:jar:2.7.2:runtime
[INFO] |  \- (xalan:serializer:jar:2.7.2:runtime - omitted for duplicate)
[INFO] \- ome:formats-api:jar:6.10.0:compile
[INFO]    +- org.openmicroscopy:ome-common:jar:6.0.9:compile
[INFO]    |  +- io.minio:minio:jar:5.0.2:compile
[INFO]    |  |  +- com.google.http-client:google-http-client-xml:jar:1.20.0:compile
[INFO]    |  |  |  +- com.google.http-client:google-http-client:jar:1.20.0:compile
[INFO]    |  |  |  |  \- (org.apache.httpcomponents:httpclient:jar:4.0.1:compile - omitted for conflict with 4.5.9)
[INFO]    |  |  |  \- xpp3:xpp3:jar:1.1.4c:compile
[INFO]    |  |  +- (com.google.guava:guava:jar:25.1-jre:compile - omitted for conflict with 28.0-jre)
[INFO]    |  |  +- com.squareup.okhttp3:okhttp:jar:3.7.0:compile
[INFO]    |  |  |  \- (com.squareup.okio:okio:jar:1.12.0:compile - omitted for duplicate)
[INFO]    |  |  +- com.squareup.okio:okio:jar:1.12.0:compile
[INFO]    |  |  +- (joda-time:joda-time:jar:2.7:compile - omitted for conflict with 2.10.3)
[INFO]    |  |  +- (com.fasterxml.jackson.core:jackson-annotations:jar:2.9.6:compile - omitted for conflict with 2.11.3)
[INFO]    |  |  +- (com.fasterxml.jackson.core:jackson-core:jar:2.9.6:compile - omitted for conflict with 2.11.3)
[INFO]    |  |  \- (com.fasterxml.jackson.core:jackson-databind:jar:2.9.6:compile - omitted for conflict with 2.11.3)
[INFO]    |  +- com.esotericsoftware:kryo:jar:4.0.2:compile
[INFO]    |  |  +- com.esotericsoftware:reflectasm:jar:1.11.3:compile
[INFO]    |  |  |  \- org.ow2.asm:asm:jar:5.0.4:compile
[INFO]    |  |  +- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO]    |  |  \- (org.sis:objenesis:jar:2.5.1:compile - omitted for conflict with 3.1)
[INFO]    |  +- (joda-time:joda-time:jar:2.2:compile - omitted for conflict with 2.10.3)
[INFO]    |  +- (com.google.guava:guava:jar:29.0-jre:compile - omitted for conflict with 28.0-jre)
[INFO]    |  +- (org.slf4j:slf4j-api:jar:1.7.6:compile - omitted for conflict with 1.7.28)
[INFO]    |  +- ch.qos.logback:logback-core:jar:1.2.9:compile
[INFO]    |  +- ch.qos.logback:logback-classic:jar:1.2.0:compile
[INFO]    |  |  +- (ch.qos.logback:logback-core:jar:1.2.0:compile - omitted for conflict with 1.2.9)
[INFO]    |  |  \- (org.slf4j:slf4j-api:jar:1.7.22:compile - omitted for conflict with 1.7.28)
[INFO]    |  +- (xalan:serializer:jar:2.7.2:runtime - omitted for duplicate)
[INFO]    |  \- (xalan:xalan:jar:2.7.2:runtime - omitted for duplicate)
[INFO]    +- org.openmicroscopy:ome-xml:jar:6.3.1:compile
[INFO]    |  +- (org.openmicroscopy:ome-common:jar:6.0.9:compile - omitted for duplicate)
[INFO]    |  +- org.openmicroscopy:specification:jar:6.3.1:compile
[INFO]    |  +- (org.slf4j:slf4j-api:jar:1.7.6:compile - omitted for conflict with 1.7.28)
[INFO]    |  \- (joda-time:joda-time:jar:2.2:compile - omitted for conflict with 2.10.3)
[INFO]    +- org.openmicroscopy:ome-codecs:jar:0.3.2:compile
[INFO]    |  +- org.openmicroscopy:ome-jai:jar:0.1.3:compile
[INFO]    |  \- (org.openmicroscopy:ome-common:jar:6.0.9:compile - omitted for duplicate)
[INFO]    +- org.slf4j:slf4j-api:jar:1.7.6:compile
[INFO]    +- (xalan:serializer:jar:2.7.2:runtime - omitted for duplicate)
[INFO]    \- (xalan:xalan:jar:2.7.2:runtime - omitted for duplicate)

So most of the dependencies which fail to resolve in the OMERO build system are defined at the low-level ome-common component and conflict with other components also defined in the the core formats-* components like cdm-core.

It's still not fully clear to me why this is an issue with this repository but not with formats-gpl which effectively pulls the same transitive dependencies and exposes them through the downstream omero-* components. My best guess it that the Ivy conflict resolution fails to resolve the dependency tree injected in https://github.com/ome/openmicroscopy/blob/develop/ivy.xml#L21.

As things stand, we have evidence that although our coupled CI workflow is sufficient to include the development ZarrReader into a development OMERO.server, this causes serious issue when trying to build OMERO in isolation. In addition to the impact on a developer workfow, a more critical impact is that release builds are likely to fail for similar reasons that caused ZarrReader to be excluded in the past. So solving this is a blocker to the inclusion of this reader into an OMERO.server release.

Trying to come up with a list of options to explore/discuss:

investigate more specifically what causes the dependency resolution to fail and fing either how to update the POM of this component and/or the Ivy settings file in OMERO to use the current injection of ZarrReader
review the low-level Bio-Formats dependencies to unify versions and minimize conflicts so that Maven, Gradle & Ivy resolve dependencies identically
update the OMERO build system to drop Ivy as the dependency resolver and use another system e.g. Gradle more compatible with the rest of the stack
migrate this reader back into the core Bio-Formats components (see also discussion in #32) and release it through omero-model etc
migrate this reader as an omero-* component using the Gradle build system

jburel commented 2 years ago

using this PR in openmicroscopy build fails with

        ::::::::::::::::::::::::::::::::::::::::::::::
        ::              FAILED DOWNLOADS            ::
        :: ^ see resolution messages for details  ^ ::
        ::::::::::::::::::::::::::::::::::::::::::::::
        :: org.objenesis#objenesis;2.5.1!objenesis.jar
        ::::::::::::::::::::::::::::::::::::::::::::::

https://github.com/ome/openmicroscopy/runs/7022030936?check_suite_focus=true

jburel commented 2 years ago

This time

        ::::::::::::::::::::::::::::::::::::::::::::::
        ::              FAILED DOWNLOADS            ::
        :: ^ see resolution messages for details  ^ ::
        ::::::::::::::::::::::::::::::::::::::::::::::
        :: junit#junit;4.12!junit.jar
        :: org.hamcrest#hamcrest-core;1.3!hamcrest-core.jar
        ::::::::::::::::::::::::::::::::::::::::::::::

If I remember correctly a bump to testng should fix that one

jburel commented 2 years ago

https://github.com/ome/openmicroscopy/runs/7023130718?check_suite_focus=true is now green with the last commit

jburel commented 2 years ago

Still green https://github.com/ome/openmicroscopy/runs/7024784119?check_suite_focus=true

ome / ZarrReader

Dependency updates #38