Multiple failures uploading to github recently

[sxa]

Seems to always be failing when it hits the mac x64 package on all versions. Rate limit? Something specific to that artifact? (Looks ok, not zero-bytes) Excluding **/*_x64_mac_* seems to progress ok (Based on one run)

Exception in thread "main" org.kohsuke.github.HttpException: Server returned HTTP response code: -1, message: 'null' for URL: https://uploads.github.com/repos/adoptium/temurin19-binaries/releases/63237181/assets?name=OpenJDK-jdk_x64_mac_hotspot_2022-03-31-03-30.pkg
    at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:494)
    at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:414)
    at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:358)
    at org.kohsuke.github.Requester.fetch(Requester.java:76)
    at org.kohsuke.github.GHRelease.uploadAsset(GHRelease.java:249)
    at org.kohsuke.github.GHRelease.uploadAsset(GHRelease.java:224)
    at net.adoptium.release.UploadAdoptReleaseFiles$_uploadFiles_closure2.doCall(UploadFiles.groovy:89)
    at net.adoptium.release.UploadAdoptReleaseFiles$_uploadFiles_closure2.call(UploadFiles.groovy)

[sxa]

It definately happens if you're only uploading the pkg files - if you filter on those it still fails.

[sophia-guo]

Issue happened on Mar 30th. Before that worked well. https://github.com/adoptium/temurin11-binaries/releases/tag/jdk11u-2022-03-29-19-34-beta

[sxa]

Note to whoever picks this up: It is not a problem if you upload a file manually to the release via the github web interface, so this is something related to the automated upload process.

[sophia-guo]

https://github.com/adoptium/github-release-scripts/issues/86#issuecomment-1084832526

Does that mean it's jenkins server related? Source code didn't change, job configuration didn't change. The only thing changed around the failure time is jenkins server has been updated. Job is running on jenkins master.

[sxa]

It's only happening with files that have a .pkg extension. If I rename a .pkg to have a .tar.gz extension it works ok, and if I take a .tar.gz and rename it to end in .pkg it fails

[sxa]

If I give a smaller (sha256.txt) file a .pkg extension it also fails, but with an HTTP/422 which kinda points to an issue with the API call:

Uploading OpenJDK18U-jdk_x64_mac_hotspot_2022-04-07-06-28.tar.gz.sha256.txt.pkg

Exception in thread "main" org.kohsuke.github.HttpException: {"message":"Validation Failed","request_id":"E072:2BF5:A6F2D4:B338F2:624EB72D","documentation_url":"https://docs.github.com/rest","errors":[{"resource":"ReleaseAsset","code":"custom","field":"content_type","message":"content_type can't be application/x-www-form-urlencoded"}]}

Caused by: java.io.IOException: Server returned HTTP response code: 422 for URL: https://uploads.github.com/repos/adoptium/temurin18-binaries/releases/63826236/assets?name=OpenJDK18U-jdk_x64_mac_hotspot_2022-04-07-06-28.tar.gz.sha256.txt.pkg

So I think we need to get some debug output from the github calls that are made to be able to progress this

[sxa]

OK So java11 is return null for the the probeContentType() method on files with a pkg extension. it should be returning a valid mime-type. java 8 works as expected and returns applicaiton/x-xar. It is unclear why this has only started happening, and it appears unrelated to the difference in OS level (Unless some fix was applied to the old one).

To replicate the problem, run this code with args[0] as one of our .pkg files: System.out.println(java.nio.file.Files.probeContentType((new java.io.File(args[0])).toPath())); (Note, it returns application/vnd.apple.installer+xml on a RHEL7 system I have access to, but everywhere else it gives null with JDK11+)

The release process is running with the JVM in /home/jenkins/.jenkins/tools/hudson.model.JDK/jdk-11.0.13_8. If I force JAVA_HOME to be a JDK8 the job works correctly. For now, I'm going to override that in script section of the the release tool jenkins job when it invokes sbin/Release.sh

Potentially related to https://github.com/adoptium/infrastructure/issues/2477 which I think whatever caused that will be what put that JDK in place.

[sxa]

It looks like JDK11 is looking exclusively at /etc/mime.types (preceded by a check in $HOME/.mime.types if present) whereas java 8 is looking in multiple locations but primarily /usr/share/mime/mime.cache. CentOS/RHEL provide an entry for pkg files in /etc/mime.types which comes from the mailcap package. On Ubuntu the file comes from the mime-support package.

The failure can therefore also be worked around by adding this line to /etc/mime.types but since that's somewhat more cyptic I don't really want to rely on this:

application/vnd.apple.installer+xml             pkg

Another option would be to add that single line to $HOME/.mime.types but i feel the forcing to JDK8 may be the less obtuse solution at the moment so i'll stick with that.

NOTE: The change that made java explicitly look at /etc/mime.types was https://github.com/openjdk/jdk11u/commit/00ad2900ddde9e7147b8cbfd7f3af77880f9397e