Java 8u292 upgrade broke Artifactory

SethTisue commented 3 years ago

jenkins-master upgraded itself to JDK 8u292 on April 24, as seen in /var/log/dpkg.log.1:

2021-04-24 06:10:30 upgrade openjdk-8-jdk:amd64 8u275-b01-1~deb9u1 8u292-b10-0+deb9u1

but Artifactory didn't start using the new JDK version until I restarted it today (I restart Artifactory once a month, with sudo systemctl restart artifactory.service because at some point we found that otherwise resources don't get freed)

and it broke. in /var/opt/jfrog/artifactory/logs/artifactory.log it said:

2021-05-14 14:13:53,519 [art-init] [WARN ] (o.j.a.c.AccessClientHttpException:41) - Unrecognized ErrorsModel by Access. 
Original message: Failed on executing /api/v1/system/ping, with response: Not Found
2021-05-14 14:13:53,520 [art-init] [ERROR] (o.a.s.a.AccessServiceImpl:395) - Could not ping access server: {}
org.jfrog.access.client.AccessClientHttpException: HTTP response status 404:Failed on executing /api/v1/system/ping, wit
h response: Not Found

some googling shows that this error can have a variety of causes. one possibility is that catalina didn't start

looking at /var/opt/jfrog/artifactory/logs/catalina/catalina.2021-05-14.log I saw:

Caused by: java.security.KeyStoreException: Key protection  algorithm not found: java.security.UnrecoverableKeyException
: Encrypt Private Key failed: unrecognized algorithm name: PBEWithSHA1AndDESede

after further googling it seems this is https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8266261

as an aside, at one point Artifactory got in a wedged state where it wouldn't start because "Artifactory Tomcat already started". got past that with netstat -ltnp | grep ':8015' and then killing the process on that port

anyway, adding -Dkeystore.pkcs12.keyProtectionAlgorithm=PBEWithHmacSHA256AndAES_256 to JAVA_OPTS in /opt/jfrog/artifactory/tomcat/bin/catalina.sh was enough for catalina to start

but artifactory.log still had the same Could not ping access server :-/

SethTisue commented 3 years ago

so that point I gave up and decided to downgrade our Java

sudo apt list -a openjdk-8-jdk only shows 252 and 292... I guess 252 is fine

the command to downgrade was sudo apt-get install openjdk-8-jdk:amd64=8u252-b09-1~deb9u1 openjdk-8-jre:amd64=8u252-b09-1~deb9u1 openjdk-8-jdk-headless:amd64=8u252-b09-1~deb9u1 openjdk-8-jre-headless:amd64=8u252-b09-1~deb9u1

OMG THAT FIXED IT :muscle::tada:

SethTisue commented 3 years ago

The same thing happened today. Same fix worked.

SethTisue commented 3 years ago

ticket about upgrading Artifactory — presumably newer versions don't have this problem: https://github.com/lightbend/scala-team/issues/175

SethTisue commented 3 years ago

our Artifactory was upgraded by @lrytz and I verified today that it now comes back up by itself after sudo systemctl restart artifactory.service

lrytz commented 3 years ago

Artifactory 7 bundles its JDK, so it no longer depends on what's installed on the system.

lrytz commented 3 years ago

(Some more details in this private repo ticket: https://github.com/lightbend/scala-team/issues/175#issuecomment-888297497)

scala / scala-jenkins-infra

Java 8u292 upgrade broke Artifactory #261