search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
2.06k stars 1.1k forks source link

S3 Object for Secondary Storage #7337

Closed tuanhoangth1603 closed 1 year ago

tuanhoangth1603 commented 1 year ago

Hello, I want to use only S3 object store for secondary. The templates of SSVM were downloaded successfully. However, SSVM cannot be started and I see the log 'No cache DataStore in scope id 1 type ZONE' and 'Can't find staging storage in zone'. Is it mandatory to use staging NFS? If so, should we make it mandatory to create staging when creating a secondary with S3?

management-server.log:

2023-03-14 11:52:12,277 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-2:ctx-d792ecfc job-2/job-27 ctx-33bf076a) (logid:bc045e3d) template 3 is already in store:1, type:Image
2023-03-14 11:52:12,285 DEBUG [o.a.c.s.d.PrimaryDataStoreImpl] (Work-Job-Executor-2:ctx-d792ecfc job-2/job-27 ctx-33bf076a) (logid:bc045e3d) Not found (templateId:3poolId:1conf:null) in template_spool_ref, persisting it
2023-03-14 11:52:12,291 DEBUG [o.a.c.s.i.TemplateDataFactoryImpl] (Work-Job-Executor-2:ctx-d792ecfc job-2/job-27 ctx-33bf076a) (logid:bc045e3d) template 3 is already in store:1, type:Primary
2023-03-14 11:52:12,295 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-2:ctx-d792ecfc job-2/job-27 ctx-33bf076a) (logid:bc045e3d) Found template routing-3 in storage pool 1 with VMTemplateStoragePool id: 1
2023-03-14 11:52:12,308 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-2:ctx-d792ecfc job-2/job-27 ctx-33bf076a) (logid:bc045e3d) Acquire lock on VMTemplateStoragePool 1 with timeout 3600 seconds
2023-03-14 11:52:12,312 INFO  [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-2:ctx-d792ecfc job-2/job-27 ctx-33bf076a) (logid:bc045e3d) lock is acquired for VMTemplateStoragePool 1
2023-03-14 11:52:12,332 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-2:ctx-d792ecfc job-2/job-27 ctx-33bf076a) (logid:bc045e3d) copyAsync inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
2023-03-14 11:52:12,344 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-2:ctx-d792ecfc job-2/job-27 ctx-33bf076a) (logid:bc045e3d) needCacheStorage true, dest at null dest role Primarytemplate/tmpl/1/3/routing-3/systemvmtemplate-4.16.1-kvm.qcow2.bz2 src role Image
2023-03-14 11:52:12,346 DEBUG [o.a.c.s.c.a.StorageCacheRandomAllocator] (Work-Job-Executor-2:ctx-d792ecfc job-2/job-27 ctx-33bf076a) (logid:bc045e3d) Can't find staging storage in zone: 1
2023-03-14 11:52:12,347 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-2:ctx-d792ecfc job-2/job-27 ctx-33bf076a) (logid:bc045e3d) copy object failed:
com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope id 1 type ZONE
        at org.apache.cloudstack.storage.cache.manager.StorageCacheManagerImpl.createCacheObject(StorageCacheManagerImpl.java:356)
        at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:166)
        at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:520)
        at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:84)
        at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:106)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:634)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:1533)
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1592)
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1698)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1201)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5853)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:6018)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2023-03-14 11:52:12,347 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-2:ctx-d792ecfc job-2/job-27 ctx-33bf076a) (logid:bc045e3d) copy failed
com.cloud.utils.exception.CloudRuntimeException: com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope id 1 type ZONE
        at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyObject(AncientDataMotionStrategy.java:214)
        at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:520)
        at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:84)
        at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:106)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createBaseImageAsync(VolumeServiceImpl.java:634)
        at org.apache.cloudstack.storage.volume.VolumeServiceImpl.createVolumeFromTemplateAsync(VolumeServiceImpl.java:1533)
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1592)
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1698)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1201)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5853)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:6018)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2023-03-14 11:52:12,359 INFO  [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-2:ctx-d792ecfc job-2/job-27 ctx-33bf076a) (logid:bc045e3d) releasing lock for VMTemplateStoragePool 1
2023-03-14 11:52:12,363 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-2:ctx-d792ecfc job-2/job-27 ctx-33bf076a) (logid:bc045e3d) Unable to create Vol[2|vm=2|ROOT]:com.cloud.utils.exception.CloudRuntimeException: com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope id 1 type ZONE
2023-03-14 11:52:12,363 INFO  [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-2:ctx-d792ecfc job-2/job-27 ctx-33bf076a) (logid:bc045e3d) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to create Vol[2|vm=2|ROOT]:com.cloud.utils.exception.CloudRuntimeException: com.cloud.utils.exception.CloudRuntimeException: No cache DataStore in scope id 1 type ZONE
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1604)
        at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1698)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1201)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5853)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:6018)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
furyflash777 commented 1 year ago

Hello! Every zone must have at least one NFS store provisioned. https://docs.cloudstack.apache.org/en/4.18.0.0/installguide/configuration.html#add-secondary-storage

mlsorensen commented 1 year ago

There's probably no reason why we couldn't use object store as a "direct download" style template where it is just pulled and cached directly to primary storage on first use. It would just need to be developed, and in particular would need to figure out how to handle authentication.

In general I think we should try to make secondary storage optional. I've used Cloudstack with direct download templates and no functioning secondary store and for the most part it works, you lose some features like volume download.