search

bytedeco / javacpp-presets

The missing Java distribution of native C++ libraries
Other
2.66k stars 741 forks source link

Nightly builds not updated atomically + old builds deleted => nightlys cannot be reliably targetted #919

Closed frankier closed 4 years ago

frankier commented 4 years ago

I don't have a lot of experience with Gradle, so the following may be problems with my own usage.

Gradle version

$ ./gradlew --version

------------------------------------------------------------
Gradle 6.5
------------------------------------------------------------

Build time:   2020-06-02 20:46:21 UTC
Revision:     a27f41e4ae5e8a41ab9b19f8dd6d86d7b384dad4

Kotlin:       1.3.72
Groovy:       2.5.11
Ant:          Apache Ant(TM) version 1.10.7 compiled on September 1 2019
JVM:          14.0.2 (Debian 14.0.2+12-Debian-1)
OS:           Linux 5.7.0-1-amd64 amd64

Nightly builds are not uploaded atomically

If we depend upon -- for example:

    compile group: "org.bytedeco", name: "openpose-platform", version: "1.6.0-1.5.4-SNAPSHOT"

When we compile:

$ ./gradlew fatJar

FAILURE: Build failed with an exception.

* Where:
Build file '/home/frankier/edu/redhen/cineast/cineast-api/build.gradle' line: 64

* What went wrong:
Could not determine the dependencies of task ':cineast-api:fatJar'.
> Could not resolve all files for configuration ':cineast-api:compile'.
   > Could not find caffe-1.0-1.5.4-SNAPSHOT-linux-x86_64.jar (org.bytedeco:caffe:1.0-1.5.4-SNAPSHOT:20200802.073134-205).
     Searched in the following locations:
         https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/caffe/1.0-1.5.4-SNAPSHOT/caffe-1.0-1.5.4-20200802.073134-205-linux-x86_64.jar
   > Could not find opencv-4.4.0-1.5.4-SNAPSHOT-linux-x86_64.jar (org.bytedeco:opencv:4.4.0-1.5.4-SNAPSHOT:20200802.070306-91).
     Searched in the following locations:
         https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/opencv/4.4.0-1.5.4-SNAPSHOT/opencv-4.4.0-1.5.4-20200802.070306-91-linux-x86_64.jar
   > Could not find hdf5-1.12.0-1.5.4-SNAPSHOT-linux-x86_64.jar (org.bytedeco:hdf5:1.12.0-1.5.4-SNAPSHOT:20200802.065104-331).
     Searched in the following locations:
         https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/hdf5/1.12.0-1.5.4-SNAPSHOT/hdf5-1.12.0-1.5.4-20200802.065104-331-linux-x86_64.jar
   > Could not find openblas-0.3.10-1.5.4-SNAPSHOT-linux-x86_64.jar (org.bytedeco:openblas:0.3.10-1.5.4-SNAPSHOT:20200802.071511-232).
     Searched in the following locations:
         https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/openblas/0.3.10-1.5.4-SNAPSHOT/openblas-0.3.10-1.5.4-20200802.071511-232-linux-x86_64.jar

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.5/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 9s

My interpretation, taking just one failure as an example: https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/caffe/1.0-1.5.4-SNAPSHOT/caffe-1.0-1.5.4-20200802.073134-205-linux-x86_64.jar cannot be found because the build for caffe-1.0-1.5.4-20200802.073134-205 is only partially finished

Proposed fix: Only add a build to https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/caffe/1.0-1.5.4-SNAPSHOT/maven-metadata.xml after it has finished building for all platforms so that gradle doesn't search for incomplete builds.

Old builds are deleted (attemped workaround)

I attempted to workaround this by pinning versions that have fully built like so:

    compile (group: "org.bytedeco", name: "openpose", version: "1.6.0-1.5.4-20200717.002050-17") { force = true }
    compile (group: "org.bytedeco", name: "hdf5", version: "1.12.0-1.5.4-20200716.042129-261") { force = true }
    compile (group: "org.bytedeco", name: "opencv", version: "4.3.0-1.5.4-20200716.042603-534") { force = true }
    compile (group: "org.bytedeco", name: "openblas", version: "0.3.10-1.5.4-20200716.042540-114") { force = true }
    compile (group: "org.bytedeco", name: "caffe", version: "1.0-1.5.4-20200717.002049-166") { force = true }
    compile (group: "org.bytedeco", name: "javacpp", version: "1.5.4-20200720.033903-461") { force = true }

This appeared to work for a while, but eventually the old builds were deleted, meaning I am only able to target the nightlies for about a week before I have to update the versions again.

saudet commented 4 years ago

Yeah, that's a known issue of Gradle that they won't fix. Please wait for the builds to finish, or use Maven: http://bytedeco.org/builds/

frankier commented 4 years ago

Thanks for your guidance. Certainly the situation is unfortunate. Oddly I went looking for an issue filed against Gradle and didn't find anything.

I'm stuck with Gradle, so this is the (platform dependent!) way I've automated the workaround (just in case anyone else stumbles upon this issue):

  1. Follow the steps towards the end of http://bytedeco.org/builds/
  2. Add the JavaCPP Gradle platform plugin: https://github.com/bytedeco/gradle-javacpp/#the-platform-plugin
  3. Add this to your build.gradle:
task fetchjavacppjars(type: Exec) {
    workingDir rootDir
    commandLine 'pwd'
    commandLine 'sh', 'fetchjavacppjars.sh', project.property("javacppPlatform")
}

compileJava.dependsOn fetchjavacppjars

fetchjavacppjars.sh:

#!/usr/bin/env sh

tmp_dir=$(mktemp -d -t mvn-XXXXXXXXXX)
cp javacppdummy.pom $tmp_dir/pom.xml
cd $tmp_dir
mvn -Djavacpp.platform=$1 --update-snapshots compile
rm -rf $tmp_dir

javacppdummy.pom (for example)

<project>
  <modelVersion>4.0.0</modelVersion>
  <groupId>i.am.a.dummy</groupId>
  <artifactId>dummy</artifactId>
  <version>0.0.1</version>
  <dependencies>
    <dependency>
      <groupId>org.bytedeco</groupId>
      <artifactId>openpose-platform</artifactId>
      <version>1.6.0-1.5.4-SNAPSHOT</version>
    </dependency>
  </dependencies>
  <repositories>
    <repository>
      <id>sonatype-nexus-snapshots</id>
      <url>https://oss.sonatype.org/content/repositories/snapshots</url>
    </repository>
  </repositories>
  <pluginRepositories>
    <pluginRepository>
      <id>sonatype-nexus-snapshots</id>
      <url>https://oss.sonatype.org/content/repositories/snapshots</url>
    </pluginRepository>
  </pluginRepositories>
</project>
saudet commented 4 years ago

Thanks for your guidance. Certainly the situation is unfortunate. Oddly I went looking for an issue filed against Gradle and didn't find anything.

See https://github.com/gradle/gradle/issues/2882

I'm stuck with Gradle, so this is the (platform dependent!) way I've automated the workaround (just in case anyone else stumbles upon this issue):

Hey, this is neat! Please consider contributing something like this for the platform plugin of Gradle JavaCPP: https://github.com/bytedeco/gradle-javacpp#the-platform-plugin It shouldn't be too hard to code everything in Groovy or Java to make this more maintainable and portable.

BTW, one of the main reasons the presets have been using classifiers is because it's not possible to have more than 1 JAR per pom.xml file without using classifiers. With Maven, that would mean maintaining 1 pom.xml file for each platform per library, which would make the whole thing a lot hairier than it already is. On the other hand, with Gradle, we can generate easily multiple pom.xml files from the same build.gradle file, so we wouldn't need to use classifiers, which would also make it possible to do things like defining platform-specific dependencies. So, if you'd like to have this really fixed, I would recommend attempting to refactor the whole build system here with Gradle since this is going to happen sooner or later.