Cloudstack Could not initialize class sun.nio.fs.UnixCopyFile

whitetiger264 commented 1 year ago

ISSUE TYPE

Bug Report

COMPONENT NAME

UI

CLOUDSTACK VERSION

4.18

CONFIGURATION

OS / ENVIRONMENT

Almalinux 8

java -version
openjdk version "11.0.19" 2023-04-18 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.19.0.7-2) (build 11.0.19+7-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.19.0.7-2) (build 11.0.19+7-LTS, mixed mode, sharing)

SUMMARY

Volume snapshots failing due to possible Java issues.

STEPS TO REPRODUCE

Create a volume snapshot of VM via UI and an error is triggered.

Here's a log entry:

2023-06-23 08:32:24,955 DEBUG [c.c.a.t.Request] (Work-Job-Executor-20:ctx-8f5395ec job-4099/job-4100 ctx-23f04f9f) (logid:931f0d86) Seq 36-8282682664687772956: Sending  { Cmd , MgmtId: 66988330791812, via: 36(hostname), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"eec8285e-8715-441b-b418-b71b231f1bab","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"85e49c98-0988-4627-984c-8f4719c401e3","name":"hostname-local-85e49c98","id":"24","poolType":"Filesystem","host":"192.168.50.7","path":"/var/lib/libvirt/images","port":"0","url":"Filesystem://192.168.50.7/var/lib/libvirt/images/?ROLE=Primary&STOREUUID=85e49c98-0988-4627-984c-8f4719c401e3","isManaged":"false"}},"name":"ROOT-120","size":"(60,00 GB) 64424509440","path":"f801a1a4-83aa-4490-a589-c2a474ec7bed","volumeId":"193","vmName":"i-2-120-VM","accountId":"2","format":"QCOW2","provisioningType":"THIN","poolId":"24","id":"193","deviceId":"0","cacheMode":"NONE","hypervisorType":"KVM","directDownload":"false","deployAsIs":"false"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"85e49c98-0988-4627-984c-8f4719c401e3","name":"hostname-local-85e49c98","id":"24","poolType":"Filesystem","host":"192.168.50.7","path":"/var/lib/libvirt/images","port":"0","url":"Filesystem://192.168.50.7/var/lib/libvirt/images/?ROLE=Primary&STOREUUID=85e49c98-0988-4627-984c-8f4719c401e3","isManaged":"false"}},"vmName":"i-2-120-VM","name":"NACDO-Backup-23-June-2023","hypervisorType":"KVM","id":"177","quiescevm":"false","physicalSize":"0"}},"wait":"0","bypassHostMaintenance":"false"}}] }
2023-06-23 08:32:25,017 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null) (logid:) Seq 36-8282682664687772956: Processing:  { Ans: , MgmtId: 66988330791812, via: 36, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NoClassDefFoundError: Could not initialize class sun.nio.fs.UnixCopyFile
    at java.base/sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:258)
    at java.base/java.nio.file.Files.copy(Files.java:1295)
    at com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.copySnapshotToPrimaryStorageDir(KVMStorageProcessor.java:1916)
    at com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.createSnapshot(KVMStorageProcessor.java:1790)
    at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:137)
    at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:65)
    at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStorageSubSystemCommandWrapper.execute(LibvirtStorageSubSystemCommandWrapper.java:36)
    at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStorageSubSystemCommandWrapper.execute(LibvirtStorageSubSystemCommandWrapper.java:30)
    at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
    at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1847)
    at com.cloud.agent.Agent.processRequest(Agent.java:662)
    at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082)
    at com.cloud.utils.nio.Task.call(Task.java:83)
    at com.cloud.utils.nio.Task.call(Task.java:29)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
","wait":"0","bypassHostMaintenance":"false"}}] }

Specific Error:

Could not initialize class sun.nio.fs.UnixCopyFile

EXPECTED RESULTS

Normal Volume snapshot processing should execute as intended.

DaanHoogland commented 1 year ago

@whitetiger264 on a running VM I get this in the UI: on the stopped vm there is no option to take a volume snapshot so I browsed to the volume and took a snapshot there. this succeeded. I seem to be using the exact same JDK/JVM on an alma 8 env. Can you explain a bit more about the scenario you are following?

whitetiger264 commented 1 year ago

@whitetiger264 on a running VM I get this in the UI: on the stopped vm there is no option to take a volume snapshot so I browsed to the volume and took a snapshot there. this succeeded. I seem to be using the exact same JDK/JVM on an alma 8 env. Can you explain a bit more about the scenario you are following?

These are the exact steps I am taking:

I go to Compute -> Instance in UI.
Stop the VM instance in question.
Then go to Storage -> Volumes in UI.
Look for the volume associated with the VM instance and select "Take Snapshot".
Give it a name such as "VMName-Backup-23-June-2023".
Select "Async" but I also tried without selecting this and then proceed to "OK".
The process is started and throws this error.

Can you replicate this, also take note of the VM name you set try making one with example-date_year

whitetiger264 commented 1 year ago

@whitetiger264 on a running VM I get this in the UI: on the stopped vm there is no option to take a volume snapshot so I browsed to the volume and took a snapshot there. this succeeded. I seem to be using the exact same JDK/JVM on an alma 8 env. Can you explain a bit more about the scenario you are following?

These are the exact steps I am taking:

I go to Compute -> Instance in UI.

Stop the VM instance in question.

Then go to Storage -> Volumes in UI.

Look for the volume associated with the VM instance and select "Take Snapshot".

Give it a name such as "VMName-Backup-23-June-2023".

Select "Async" but I also tried without selecting this and then proceed to "OK".

The process is started and throws this error.

Can you replicate this, also take note of the VM name you set try making one with example-date_year

I tried it now again. Two things I did differently:

Restarted cloudstack-management, cloudstack-usage and cloudstack-agent. So that new Java processes are created.
Changed the volume snapshot name by not using the - and _ characters and no spaces. And now it magically works:

whitetiger264 commented 1 year ago

I tried another one now and used the - character as before, it now works perfectly fine. The reason I rebooted cloudstack-management, cloudstack-usage and cloudstack-agent was because of this guide:

https://github.com/CatalogueOfLife/backend/issues/621

Specifically this part:

a very weird error. I have restarted the server with the exact same jar and it's fine now. I can upload archives just fine

But I restarted the services instead of the server.

Someone then says the following below that thread:

This was probably caused when a Java security update was applied by CentOS, but running Java processes wasn't restarted.

This is most likely the same reason for Almalinux in my case. Since I can now no longer replicate this error, this report can be closed. Thank you.

apache / cloudstack