Linstor VM/Volume snapshots

[UAnton]

ISSUE TYPE

• Other

COMPONENT NAME

Storage

CLOUDSTACK VERSION

4.19.0.2

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

How can I create snapshots/backups?

STEPS TO REPRODUCE

Try to create a VM/Volume snapshot

EXPECTED RESULTS

Snapshot created

ACTUAL RESULTS

get error - Error while taking snapshot

[UAnton]

2024-07-09 10:18:15,953 ERROR [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-4:ctx-753b7118 job-52 ctx-7d1d55c9) (logid:feb78585) Failed to create snapshot due to an internal error creating snapshot for volume 12bf6956-591f-4f53-a1a1-36cb932752e0
java.lang.RuntimeException: Unexpected exception
        at com.cloud.storage.VolumeApiServiceImpl.takeSnapshotInternal(VolumeApiServiceImpl.java:3528)
        at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:3445)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy236.takeSnapshot(Unknown Source)
        at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:228)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to unhide zfs snapshot device.
        at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:382)
        at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:193)
        at com.cloud.storage.snapshot.SnapshotManagerImpl.backupSnapshotToSecondary(SnapshotManagerImpl.java:1439)
        ... 45 more
2024-07-09 10:18:15,955 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-4:ctx-753b7118 job-52) (logid:feb78585) Complete async job-52, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 12bf6956-591f-4f53-a1a1-36cb932752e0"}

[DaanHoogland]

I see I made some assumption that might not be true. Is this a linstor environment @UAnton ? what storage type(s) did you try this with?

(I was looking at the title, so maybe not so wrong after all)

can you have a look as well @rp- ?

[UAnton]

@DaanHoogland I use Linstor. When I try to take a snapshot, I get an error in the UI and don’t see the snapshot, but the snapshot is created in the Linstor storage

[rp-]

Well the exception kinda tells what the problem is: Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to unhide zfs snapshot device.

We copy the data to secondary storage, from the zfs snapshot. But zfs snapshots are not directly accessible by default and you have to unhide the block device. For some reason the unhide fails on your setup. Can you try to manually unhide such a snapshot to see the error? e.g.: zfs set snapdev=visible <zfs-snapshotname>

Or we need the agent debug output of the failed request.