search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
2.06k stars 1.1k forks source link

Exceptions in cloud.log related to VMware, while uploading a KVM template/iso "From local" #8376

Closed andrijapanicsb closed 4 months ago

andrijapanicsb commented 10 months ago

ACS 4.18.1

Even though the upload of the QCOW2 in a pure KVM env works - there are exceptions/NPE in the cloud.log inside SSVM that mistakenly looks for VMware disks etc. This is "Upload from local:"

2023-12-19 09:41:01,240 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) BODY FileUpload: FileUpload: content-disposition: form-data; name="files[]"; filename="**jammy-server-cloudimg-arm64.img**"
content-type: application/octet-stream; charset=UTF-8
content-length: 652804096
Completed: true
IsInMemory: false
RealFile: /mnt/SecStorage/8c7e52fe-4c67-3a35-866d-fad343722f12/template/tmpl/2/205/FUp_16377585739507701743_upload DefaultDeleteAfter: true
2023-12-19 09:41:01,779 INFO  [storage.template.OVAProcessor] (nioEventLoopGroup-3-1:null) Template processing. templatePath: /mnt/SecStorage/8c7e52fe-4c67-3a35-866d-fad343722f12/template/tmpl/2/205, templateName: 2c97c478-b4ae-39ec-acdb-0ae2df36a203
2023-12-19 09:41:01,783 INFO  [storage.template.OVAProcessor] (nioEventLoopGroup-3-1:null) Unable to find the vmware template file: /mnt/SecStorage/8c7e52fe-4c67-3a35-866d-fad343722f12/template/tmpl/2/205/2c97c478-b4ae-39ec-acdb-0ae2df36a203.ova
2023-12-19 09:41:01,784 INFO  [storage.template.VmdkProcessor] (nioEventLoopGroup-3-1:null) Template processing. templatePath: /mnt/SecStorage/8c7e52fe-4c67-3a35-866d-fad343722f12/template/tmpl/2/205, templateName: 2c97c478-b4ae-39ec-acdb-0ae2df36a203
2023-12-19 09:41:01,787 INFO  [storage.template.VmdkProcessor] (nioEventLoopGroup-3-1:null) Unable to find the vmware template file: /mnt/SecStorage/8c7e52fe-4c67-3a35-866d-fad343722f12/template/tmpl/2/205/2c97c478-b4ae-39ec-acdb-0ae2df36a203.vmdk
2023-12-19 09:41:02,100 WARN  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) 
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
    at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
    at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
    at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
    at io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:284)
    at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:952)
    at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:935)
    at io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:247)
    at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232)
    at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222)
    at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
    at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)
2023-12-19 09:41:02,126 WARN  [netty.channel.AbstractChannelHandlerContext] (nioEventLoopGroup-3-1:null) An exception 'java.lang.NullPointerException' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
    at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
    at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
    at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
    at io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:284)
    at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:952)
    at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:935)
    at io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:247)
    at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232)
    at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222)
    at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
    at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)
andrijapanicsb commented 10 months ago

The same happens when uploading ISO "From local" - with the builtin SSL (no special SSL config needed at all in the) - just open the "redirect" URL in another tab, accept the SSL untrusted exception, and then try to "Upload from local"

2023-12-19 09:43:32,054 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-3:null) BODY FileUpload: FileUpload: content-disposition: form-data; name="files[]"; filename="ubuntu-22.04.3-live-server-amd64 (1).iso"
content-type: application/octet-stream; charset=UTF-8
content-length: 2133391360
Completed: true
IsInMemory: false
RealFile: /mnt/SecStorage/f099220e-c7d5-35e9-b344-60b0f41fbce0/template/tmpl/2/225/FUp_4105661296274340865_upload DefaultDeleteAfter: true
2023-12-19 09:43:32,357 INFO  [storage.template.OVAProcessor] (nioEventLoopGroup-3-3:null) Template processing. templatePath: /mnt/SecStorage/f099220e-c7d5-35e9-b344-60b0f41fbce0/template/tmpl/2/225, templateName: 82f296cd-b87a-4467-bd93-4018682a9911
2023-12-19 09:43:32,359 INFO  [storage.template.OVAProcessor] (nioEventLoopGroup-3-3:null) Unable to find the vmware template file: /mnt/SecStorage/f099220e-c7d5-35e9-b344-60b0f41fbce0/template/tmpl/2/225/82f296cd-b87a-4467-bd93-4018682a9911.ova
2023-12-19 09:43:32,360 INFO  [storage.template.VmdkProcessor] (nioEventLoopGroup-3-3:null) Template processing. templatePath: /mnt/SecStorage/f099220e-c7d5-35e9-b344-60b0f41fbce0/template/tmpl/2/225, templateName: 82f296cd-b87a-4467-bd93-4018682a9911
2023-12-19 09:43:32,362 INFO  [storage.template.VmdkProcessor] (nioEventLoopGroup-3-3:null) Unable to find the vmware template file: /mnt/SecStorage/f099220e-c7d5-35e9-b344-60b0f41fbce0/template/tmpl/2/225/82f296cd-b87a-4467-bd93-4018682a9911.vmdk
2023-12-19 09:43:32,416 WARN  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-3:null) 
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
    at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
    at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
    at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
    at io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:284)
    at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:952)
    at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:935)
    at io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:247)
    at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232)
    at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222)
    at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
    at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)
2023-12-19 09:43:32,419 WARN  [netty.channel.AbstractChannelHandlerContext] (nioEventLoopGroup-3-3:null) An exception 'java.lang.NullPointerException' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
    at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
    at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
    at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:76)
    at io.netty.handler.codec.http.multipart.DefaultHttpDataFactory.cleanRequestHttpData(DefaultHttpDataFactory.java:284)
    at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.cleanFiles(HttpPostMultipartRequestDecoder.java:952)
    at io.netty.handler.codec.http.multipart.HttpPostMultipartRequestDecoder.destroy(HttpPostMultipartRequestDecoder.java:935)
    at io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:247)
    at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.reset(HttpUploadServerHandler.java:232)
    at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:222)
    at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:302)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
    at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)
DaanHoogland commented 10 months ago

yes disturbing, I think a warn should not add a stacktrace. you can turn on debug for that.

DaanHoogland commented 4 months ago

fixed by #9220