scylla-jmx.service: failed during artifact tests

[Annamikhlin]

Some artifact tests are failed with:

09:01:19  Command: '/usr/bin/nodetool  status '
09:01:19  Exit code: 1
09:01:19  Stdout:
09:01:19  Stderr:
09:01:19  nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.

Related to

2023-04-16T05:59:02.939 artifacts-rocky8-jenkins-db-node-bcd77d4c-0-1 !NOTICE | systemd[1]: scylla-jmx.service: Main process exited, code=exited, status=1/FAILURE
2023-04-16T05:59:02.939 artifacts-rocky8-jenkins-db-node-bcd77d4c-0-1 !WARNING | systemd[1]: scylla-jmx.service: Failed with result 'exit-code'.

links to failed jobs: https://jenkins.scylladb.com/job/scylla-master/job/artifacts/job/artifacts-ubuntu2004-arm-test/370/ https://jenkins.scylladb.com/job/scylla-master/job/artifacts/job/artifacts-ubuntu2004-test/929/ https://jenkins.scylladb.com/job/scylla-master/job/artifacts/job/artifacts-rocky8-test/209/

[Annamikhlin]

@fgelcer - please add the relevant logs

/cc: @fruch

[fruch]

the actual failure (from ubuntu2004):

Apr 16 07:31:45 artifacts-ubuntu2004-jenkins-db-node-7d39823b-0-1 scylla-jmx[20387]:  using Hotspot JIT.
Apr 16 07:31:45 artifacts-ubuntu2004-jenkins-db-node-7d39823b-0-1 scylla-jmx[20387]:  .
Apr 16 07:31:45 artifacts-ubuntu2004-jenkins-db-node-7d39823b-0-1 scylla-jmx[20387]:  The packages are built using the IcedTea build support and patches
Apr 16 07:31:45 artifacts-ubuntu2004-jenkins-db-node-7d39823b-0-1 scylla-jmx[20387]:  from the IcedTea project.
Apr 16 07:31:45 artifacts-ubuntu2004-jenkins-db-node-7d39823b-0-1 scylla-jmx[20387]: Homepage: http://openjdk.java.net/
Apr 16 07:31:45 artifacts-ubuntu2004-jenkins-db-node-7d39823b-0-1 scylla-jmx[20387]: Original-Maintainer: Java Maintenance <debian-java@lists.debian.org>
Apr 16 07:31:45 artifacts-ubuntu2004-jenkins-db-node-7d39823b-0-1 scylla-jmx[20384]: Picked up JAVA_TOOL_OPTIONS:
Apr 16 07:31:45 artifacts-ubuntu2004-jenkins-db-node-7d39823b-0-1 scylla-jmx[20384]: Error: Could not find or load main class .usr.lib.jvm.java-8-openjdk-amd64.bin.java

and on equivalent ARM:

Apr 16 07:33:21 artifacts-ubuntu2004-jenkins-db-node-e9f56533-1 scylla-jmx[40744]:  using Hotspot JIT.
Apr 16 07:33:21 artifacts-ubuntu2004-jenkins-db-node-e9f56533-1 scylla-jmx[40744]:  .
Apr 16 07:33:21 artifacts-ubuntu2004-jenkins-db-node-e9f56533-1 scylla-jmx[40744]:  The packages are built using the IcedTea build support and patches
Apr 16 07:33:21 artifacts-ubuntu2004-jenkins-db-node-e9f56533-1 scylla-jmx[40744]:  from the IcedTea project.
Apr 16 07:33:21 artifacts-ubuntu2004-jenkins-db-node-e9f56533-1 scylla-jmx[40744]: Homepage: http://openjdk.java.net/
Apr 16 07:33:21 artifacts-ubuntu2004-jenkins-db-node-e9f56533-1 scylla-jmx[40744]: Original-Maintainer: Java Maintenance <debian-java@lists.debian.org>
Apr 16 07:33:21 artifacts-ubuntu2004-jenkins-db-node-e9f56533-1 scylla-jmx[40741]: Picked up JAVA_TOOL_OPTIONS:
Apr 16 07:33:21 artifacts-ubuntu2004-jenkins-db-node-e9f56533-1 scylla-jmx[40741]: Error: Could not find or load main class .usr.lib.jvm.java-8-openjdk-arm64.bin.java

[yaronkaikov]

/cc @DoronArazii Please track this as well

[fgelcer]

here the logs of one of the failed runs:

How frequently does it reproduce?

every run on Ubuntu20 (x86_64 and ARM)

Installation details

Cluster size: 1 nodes (n1-standard-2)

Scylla Nodes used in this run:

• artifacts-ubuntu2004-jenkins-db-node-7d39823b-0-1 (35.229.51.124 | 10.142.0.75) (shards: 2)

OS / Image: https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/family/ubuntu-2004-lts (gce: us-east1)

Test: artifacts-ubuntu2004-test Test id: 7d39823b-7877-4409-94b9-7ca4e262a33f Test name: scylla-master/artifacts/artifacts-ubuntu2004-test Test config file(s):

• ubuntu2004.yaml

Logs and commands

- Restore Monitor Stack command: `$ hydra investigate show-monitor 7d39823b-7877-4409-94b9-7ca4e262a33f` - Restore monitor on AWS instance using [Jenkins job](https://jenkins.scylladb.com/view/QA/job/QA-tools/job/hydra-show-monitor/parambuild/?test_id=7d39823b-7877-4409-94b9-7ca4e262a33f) - Show all stored logs command: `$ hydra investigate show-logs 7d39823b-7877-4409-94b9-7ca4e262a33f` ## Logs: - **db-cluster-7d39823b.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/7d39823b-7877-4409-94b9-7ca4e262a33f/20230416_073326/db-cluster-7d39823b.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/7d39823b-7877-4409-94b9-7ca4e262a33f/20230416_073326/db-cluster-7d39823b.tar.gz) - **sct-runner-events-7d39823b.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/7d39823b-7877-4409-94b9-7ca4e262a33f/20230416_073326/sct-runner-events-7d39823b.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/7d39823b-7877-4409-94b9-7ca4e262a33f/20230416_073326/sct-runner-events-7d39823b.tar.gz) - **sct-7d39823b.log.tar.gz** - [https://cloudius-jenkins-test.s3.amazonaws.com/7d39823b-7877-4409-94b9-7ca4e262a33f/20230416_073326/sct-7d39823b.log.tar.gz](https://cloudius-jenkins-test.s3.amazonaws.com/7d39823b-7877-4409-94b9-7ca4e262a33f/20230416_073326/sct-7d39823b.log.tar.gz) [Jenkins job URL](https://jenkins.scylladb.com/job/scylla-master/job/artifacts/job/artifacts-ubuntu2004-test/929/)

[DoronArazii]

@yaronkaikov / @fgelcer is it blocking something? (versions wise)

[yaronkaikov]

@yaronkaikov / @fgelcer is it blocking something? (versions wise)

All artifacts are failing on master , so I would say it's important

[DoronArazii]

Will add it to our daily status meeting (master area)

[tchaikov]

it does seem a regression due to recent change moving to OpenJDK-11. i failed to come up with root cause analysis today, will continue the investigation tomorrow.