search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
1.98k stars 1.09k forks source link

Getting Error 502 at the end of ISO upload under the Images menu #8175

Open rongage opened 10 months ago

rongage commented 10 months ago
ISSUE TYPE
COMPONENT NAME

UI and possibly management server

CLOUDSTACK VERSION

4.18.1.0

CONFIGURATION

Simple beginners install. Management Server on CentOS 9 (yeah, I know, not supported), KVM Host on Ubuntu 22.04 LTS. NFS for both primary and secondary storage hosted on management server.

OS / ENVIRONMENT
SUMMARY

Uploading of an ISO fails after seemingly going through the entire upload process (progress bar go to 100% as expected), then error message in upper right corner informing me that the upload failed due to status code 502.

STEPS TO REPRODUCE

Log in as admin navigate to Images -> ISOs Upload an ISO file (I tried both a Windows 11 ISO as well as Alma Linux 9 ISO). When progress bar hits 100%, the status 502 error comes up and the upload never "finishes"

EXPECTED RESULTS

Uploaded file eventually becomes available under Images->ISOs

ACTUAL RESULTS

As stated above, I get a status 502 error after the upload finishes.

Note that I tried turning off the config item Use.HTTPS.To.Upload and that does not make any difference.

DaanHoogland commented 10 months ago

@rongage centos9 may not be supported but rocky9, alma9 and oracle9 are, so I'll test on one of these and see if I can reproduce.

rongage commented 10 months ago

I found this gem in the logs on the SSVM:

2023-11-04 17:52:18,835 INFO [handler.logging.LoggingHandler] (nioEventLoopGroup-2-1:null) [id: 0xe4c09b33, L:/0.0.0.0:8210] READ: [id: 0x9b0f4dd7, L:/127.0.0.1:8210 - R:/127.0.0.1:42274] 2023-11-04 17:52:18,838 INFO [handler.logging.LoggingHandler] (nioEventLoopGroup-2-1:null) [id: 0xe4c09b33, L:/0.0.0.0:8210] READ COMPLETE 2023-11-04 17:52:18,868 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-2:null) HEADER: SIGNATURE=yYhTN548vSYA3hxLoDUoO1exMpo= 2023-11-04 17:52:18,869 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-2:null) HEADER: METADATA=8y9bShbEVTuZDKwp0YQhnndARQiJvXt0FOdKo69WPE9dbEHUGAGU24IbuiUTALNWPzgTIMcgqT37V8R3U5eCGIlxBHZR21Dyx3V32sJwZcGPNT6cxXUr+ELrFDvC/1O7O8g+GqxI0umLOTPLfIKegkh9e1R1tr9PmirgOcjl8WhzpmLN9d3oyKmeJjjF+YaL/XELK4+mZ++rjb9q1OdQz/hFDqCa4mXp8fTbY6DFx1/K9k7Lfxmr0bUeuWbZ23DZSk/3VC1BTy4wgSJ2HwR6m0a2jV5NllLkfLxZm9WFkSF8nkF18P/ntWcDwh5eg6zOoBy5uEwiIEJiF0VaN5qUncRE8K7gttJFeAVKYUAzGZUafv9Im0gbM3AQxt+rDBMLa8y/NwV1wCy8PTYGqirCbzgtGpVOfKFZ2WeGBX8lqdR/pxAZLZCjKBRo++KsAoXndqOqYnPea+rP5/7O8y00uVSSIW/dfhTjtiRi9oOBUdTQZF2vcEh0kXUJCeahtXE8Vgi85FriTf7u/4DR31mI5EBpqlI/mKqLwsvEweYLYsNDQoEfVi9HBBSYQbvmbIvd3kOD+fR+If0QrTBzm5eWxDKnIxdFp4wtBrSAUp+cfQ== 2023-11-04 17:52:18,870 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-2:null) HEADER: EXPIRES=2023-11-04T18:07:18.759Z 2023-11-04 17:52:18,870 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-2:null) HEADER: HOST=192.168.0.152 2023-11-04 17:52:18,871 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-2:null) HEADER: CONTENT_LENGTH=10653532389 2023-11-04 17:52:18,871 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-2:null) URI: uuid=0d9e8931-2a5b-408a-a84c-e138e798626a 2023-11-04 17:52:18,872 ERROR [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-2:null) post request validation failed com.cloud.exception.InvalidParameterValueException: signature validation failed. at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.validatePostUploadRequest(NfsSecondaryStorageResource.java:3536) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:165) at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:315) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:429) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829) 2023-11-04 17:52:41,401 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-1:null) Determined host 192.168.0.141 corresponds to IP 192.168.0.141 2023-11-04 17:53:41,711 INFO [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-3:null) Determined host 192.168.0.141 corresponds to IP 192.168.0.141

DaanHoogland commented 10 months ago

Ah, this has to do with certificates. Uploading from local will fail this way. The trick is to try an

let me know if this works @rongage

rongage commented 10 months ago

That is the first thing I tried when this started messing up (I was able to upload 3 ISOs before it stopped working).

Apparently Chrome has memorized the cert for the SSVM because browsing to the IP address - in my case: https://192.168.0.152 - does not present an "accept this cert" dialog.  In fact, if this was the problem, then wouldn't the upload fail to even start (I get the upload progress dialog without a problem).

I had found similar issues to this one on GitHub (#7442, #7981). The resolution to that issue mentioned something about a firewall rule to the SSVM.  For grins, I tried that and no difference was encountered (or I misunderstood where to modify the firewall rule).

Any chance for other things to try?

Ron Gage

On 11/6/2023 3:20 AM, dahn wrote:

Ah, this has to do with certificates. Uploading from local will fail this way. The trick is to try an

  • upload while having the debugger/network log open in your browser
  • get the actual upload url
  • open another window
  • go to the base url of the upload
  • accept the server cert
  • go back and try to upload

let me know if this works @rongage https://github.com/rongage

— Reply to this email directly, view it on GitHub https://github.com/apache/cloudstack/issues/8175#issuecomment-1794291710, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALT2YJCUMUAQVNQCHZXCPADYDCMWHAVCNFSM6AAAAAA62DAMISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJUGI4TCNZRGA. You are receiving this because you were mentioned.Message ID: @.***>

weizhouapache commented 10 months ago

That is the first thing I tried when this started messing up (I was able to upload 3 ISOs before it stopped working).

Apparently Chrome has memorized the cert for the SSVM because browsing to the IP address - in my case: https://192.168.0.152 - does not present an "accept this cert" dialog.  In fact, if this was the problem, then wouldn't the upload fail to even start (I get the upload progress dialog without a problem).

I had found similar issues to this one on GitHub (#7442, #7981). The resolution to that issue mentioned something about a firewall rule to the SSVM.  For grins, I tried that and no difference was encountered (or I misunderstood where to modify the firewall rule).

Any chance for other things to try?

Ron Gage

On 11/6/2023 3:20 AM, dahn wrote:

Ah, this has to do with certificates. Uploading from local will fail this way. The trick is to try an

  • upload while having the debugger/network log open in your browser
  • get the actual upload url
  • open another window
  • go to the base url of the upload
  • accept the server cert
  • go back and try to upload

let me know if this works @rongage https://github.com/rongage

— Reply to this email directly, view it on GitHub https://github.com/apache/cloudstack/issues/8175#issuecomment-1794291710, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALT2YJCUMUAQVNQCHZXCPADYDCMWHAVCNFSM6AAAAAA62DAMISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJUGI4TCNZRGA. You are receiving this because you were mentioned.Message ID: @.***>

Can you check if there are errors in /var/log/cloud.log in SSVM?

rongage commented 10 months ago

Just tried uploading an ISO again.  This exception throws immediately at the start of the upload:

2023-11-06 23:25:48,087 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: SIGNATURE=ufQC9ZETkswaVrCh78Bps4T7kfw= 2023-11-06 23:25:48,088 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: METADATA=Kks2KPhSwMltHKEpFTOvS8rR1gcdyiG6eqciFzztBm1YUip/FEkycNf0X8BXEDJYBIg8DrILrhlxuT+KPn8RejFOjyVn5TTrgYVyhhC0j8Wk6NZ/m70VbUjJ6EUj+iRnjrrutKBxs+E6a+l4qhL7kOavy6yvG1+c426wByjQIVUcyCRbpnnzygIh/nZrxsynH5jjxibhfqTjDqjNLPUV4eorbPAtmEr20gGn8RGtf7+rdaX4cJPUKDaKYoESUcbKezbtreDfcl9BPJuyrZ9rd/6KaG5YCrInaf58yMxMDtZOZMzZ0HB4Ln5nIE/LwBhQIaHZ/o+LzC/j1FXPYkDwbEzENYzosCUTXJBMWcLn3v16hjhUJockoE8rc1N/A5P8KdBronQ/PGwgSW8e0d1vDbRvVwkzduagCadcRTbub1GBVhCKphak/2prUt6Z3Kn5sJNoGCvT0NManLp0QV2j/5DtB+NSQZLnn+CEf6johZ4B044BqRlWAV7CDPOR8ME9BhPjDCfdCkr7V+lvMYnUgsyxdihLsgeYU4WZk6xYfLfJAjo0FJQbJusaf7jsz2Yh6jQPtfd+F3Lni+1GWc9MQH3czJ8qtISkse0= 2023-11-06 23:25:48,089 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: EXPIRES=2023-11-06T23:40:47.341Z 2023-11-06 23:25:48,090 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: HOST=192.168.0.152 2023-11-06 23:25:48,090 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: CONTENT_LENGTH=5372111111 2023-11-06 23:25:48,092 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) URI: uuid=67f72e90-a981-4d76-a7ea-03a619d1172d 2023-11-06 23:25:48,147 ERROR [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) post request validation failed com.cloud.exception.InvalidParameterValueException: signature validation failed.         at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.validatePostUploadRequest(NfsSecondaryStorageResource.java:3536)         at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:165)         at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)         at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)         at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)         at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:315)         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:429)         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)         at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)         at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)         at java.base/java.lang.Thread.run(Thread.java:829)

Upload progress dialog continues after the exception is thrown.

Ron

On 11/6/2023 6:14 PM, Wei Zhou wrote:

That is the first thing I tried when this started messing up (I
was able
to upload 3 ISOs before it stopped working).

Apparently Chrome has memorized the cert for the SSVM because browsing
to the IP address - in my case: https://192.168.0.152 - does not
present
an "accept this cert" dialog.  In fact, if this was the problem, then
wouldn't the upload fail to even start (I get the upload progress
dialog
without a problem).

I had found similar issues to this one on GitHub (#7442
<https://github.com/apache/cloudstack/issues/7442>, #7981
<https://github.com/apache/cloudstack/issues/7981>). The
resolution to that issue mentioned something about a firewall rule to
the SSVM.  For grins, I tried that and no difference was
encountered (or
I misunderstood where to modify the firewall rule).

Any chance for other things to try?

Ron Gage

On 11/6/2023 3:20 AM, dahn wrote:

    Ah, this has to do with certificates. Uploading from local
    will fail
    this way. The trick is to try an

      * upload while having the debugger/network log open in your
        browser
      * get the actual upload url
      * open another window
      * go to the base url of the upload
      * accept the server cert
      * go back and try to upload

    let me know if this works @rongage
    <https://github.com/rongage> https://github.com/rongage

    —
    Reply to this email directly, view it on GitHub
    #8175 (comment)
    <https://github.com/apache/cloudstack/issues/8175#issuecomment-1794291710>,
    or unsubscribe
    https://github.com/notifications/unsubscribe-auth/ALT2YJCUMUAQVNQCHZXCPADYDCMWHAVCNFSM6AAAAAA62DAMISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJUGI4TCNZRGA.
    You are receiving this because you were mentioned.Message ID:
    /*@*/.***>

Can you check if there are errors in /var/log/cloud.log in SSVM?

— Reply to this email directly, view it on GitHub https://github.com/apache/cloudstack/issues/8175#issuecomment-1797012554, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALT2YJCJBY3YQMDMBUNYNELYDFVOLAVCNFSM6AAAAAA62DAMISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJXGAYTENJVGQ. You are receiving this because you were mentioned.Message ID: @.***>

rongage commented 10 months ago

I did just now try with Edge instead of Chrome - same result.

I also tried re-enabling https on the upload setting (use.https.to.upload) - That only prevents the progress dialog from coming up.

I also did the network call trace for the upload.  The erroring URL is: https://192.168.0.152/upload/5e13667c-a2e0-4329-9752-0529676a88b4 Trying to open that URL does do anything.  The only thing that comes out of that is that the error point in the UI code is xhr.js:187.  No response was received from that URL call.

Is it at all possible that this URL should be http instead of https since I have the https setting off.

Ron Gage

On 11/6/2023 6:28 PM, Ron Gage wrote:

Just tried uploading an ISO again.  This exception throws immediately at the start of the upload:

2023-11-06 23:25:48,087 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: SIGNATURE=ufQC9ZETkswaVrCh78Bps4T7kfw= 2023-11-06 23:25:48,088 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: METADATA=Kks2KPhSwMltHKEpFTOvS8rR1gcdyiG6eqciFzztBm1YUip/FEkycNf0X8BXEDJYBIg8DrILrhlxuT+KPn8RejFOjyVn5TTrgYVyhhC0j8Wk6NZ/m70VbUjJ6EUj+iRnjrrutKBxs+E6a+l4qhL7kOavy6yvG1+c426wByjQIVUcyCRbpnnzygIh/nZrxsynH5jjxibhfqTjDqjNLPUV4eorbPAtmEr20gGn8RGtf7+rdaX4cJPUKDaKYoESUcbKezbtreDfcl9BPJuyrZ9rd/6KaG5YCrInaf58yMxMDtZOZMzZ0HB4Ln5nIE/LwBhQIaHZ/o+LzC/j1FXPYkDwbEzENYzosCUTXJBMWcLn3v16hjhUJockoE8rc1N/A5P8KdBronQ/PGwgSW8e0d1vDbRvVwkzduagCadcRTbub1GBVhCKphak/2prUt6Z3Kn5sJNoGCvT0NManLp0QV2j/5DtB+NSQZLnn+CEf6johZ4B044BqRlWAV7CDPOR8ME9BhPjDCfdCkr7V+lvMYnUgsyxdihLsgeYU4WZk6xYfLfJAjo0FJQbJusaf7jsz2Yh6jQPtfd+F3Lni+1GWc9MQH3czJ8qtISkse0= 2023-11-06 23:25:48,089 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: EXPIRES=2023-11-06T23:40:47.341Z 2023-11-06 23:25:48,090 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: HOST=192.168.0.152 2023-11-06 23:25:48,090 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: CONTENT_LENGTH=5372111111 2023-11-06 23:25:48,092 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) URI: uuid=67f72e90-a981-4d76-a7ea-03a619d1172d 2023-11-06 23:25:48,147 ERROR [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) post request validation failed com.cloud.exception.InvalidParameterValueException: signature validation failed.         at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.validatePostUploadRequest(NfsSecondaryStorageResource.java:3536)         at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:165)         at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)         at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)         at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)         at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:315)         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:429)         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)         at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)         at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)         at java.base/java.lang.Thread.run(Thread.java:829)

Upload progress dialog continues after the exception is thrown.

Ron

On 11/6/2023 6:14 PM, Wei Zhou wrote:

That is the first thing I tried when this started messing up (I
was able
to upload 3 ISOs before it stopped working).

Apparently Chrome has memorized the cert for the SSVM because
browsing
to the IP address - in my case: https://192.168.0.152 - does not
present
an "accept this cert" dialog.  In fact, if this was the problem, then
wouldn't the upload fail to even start (I get the upload progress
dialog
without a problem).

I had found similar issues to this one on GitHub (#7442
<https://github.com/apache/cloudstack/issues/7442>, #7981
<https://github.com/apache/cloudstack/issues/7981>). The
resolution to that issue mentioned something about a firewall rule to
the SSVM.  For grins, I tried that and no difference was
encountered (or
I misunderstood where to modify the firewall rule).

Any chance for other things to try?

Ron Gage

On 11/6/2023 3:20 AM, dahn wrote:

    Ah, this has to do with certificates. Uploading from local
    will fail
    this way. The trick is to try an

      * upload while having the debugger/network log open in your
        browser
      * get the actual upload url
      * open another window
      * go to the base url of the upload
      * accept the server cert
      * go back and try to upload

    let me know if this works @rongage
    <https://github.com/rongage> https://github.com/rongage

    —
    Reply to this email directly, view it on GitHub
    #8175 (comment)
    <https://github.com/apache/cloudstack/issues/8175#issuecomment-1794291710>,
    or unsubscribe
    https://github.com/notifications/unsubscribe-auth/ALT2YJCUMUAQVNQCHZXCPADYDCMWHAVCNFSM6AAAAAA62DAMISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJUGI4TCNZRGA.
    You are receiving this because you were mentioned.Message ID:
    /*@*/.***>

Can you check if there are errors in /var/log/cloud.log in SSVM?

— Reply to this email directly, view it on GitHub https://github.com/apache/cloudstack/issues/8175#issuecomment-1797012554, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALT2YJCJBY3YQMDMBUNYNELYDFVOLAVCNFSM6AAAAAA62DAMISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJXGAYTENJVGQ. You are receiving this because you were mentioned.Message ID: @.***>

rongage commented 10 months ago

Strike that last line about https.  I don't know what I am doing sometimes...

Ron

On 11/6/2023 7:06 PM, Ron Gage wrote:

I did just now try with Edge instead of Chrome - same result.

I also tried re-enabling https on the upload setting (use.https.to.upload) - That only prevents the progress dialog from coming up.

I also did the network call trace for the upload.  The erroring URL is: https://192.168.0.152/upload/5e13667c-a2e0-4329-9752-0529676a88b4 Trying to open that URL does do anything.  The only thing that comes out of that is that the error point in the UI code is xhr.js:187.  No response was received from that URL call.

Is it at all possible that this URL should be http instead of https since I have the https setting off.

Ron Gage

On 11/6/2023 6:28 PM, Ron Gage wrote:

Just tried uploading an ISO again.  This exception throws immediately at the start of the upload:

2023-11-06 23:25:48,087 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: SIGNATURE=ufQC9ZETkswaVrCh78Bps4T7kfw= 2023-11-06 23:25:48,088 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: METADATA=Kks2KPhSwMltHKEpFTOvS8rR1gcdyiG6eqciFzztBm1YUip/FEkycNf0X8BXEDJYBIg8DrILrhlxuT+KPn8RejFOjyVn5TTrgYVyhhC0j8Wk6NZ/m70VbUjJ6EUj+iRnjrrutKBxs+E6a+l4qhL7kOavy6yvG1+c426wByjQIVUcyCRbpnnzygIh/nZrxsynH5jjxibhfqTjDqjNLPUV4eorbPAtmEr20gGn8RGtf7+rdaX4cJPUKDaKYoESUcbKezbtreDfcl9BPJuyrZ9rd/6KaG5YCrInaf58yMxMDtZOZMzZ0HB4Ln5nIE/LwBhQIaHZ/o+LzC/j1FXPYkDwbEzENYzosCUTXJBMWcLn3v16hjhUJockoE8rc1N/A5P8KdBronQ/PGwgSW8e0d1vDbRvVwkzduagCadcRTbub1GBVhCKphak/2prUt6Z3Kn5sJNoGCvT0NManLp0QV2j/5DtB+NSQZLnn+CEf6johZ4B044BqRlWAV7CDPOR8ME9BhPjDCfdCkr7V+lvMYnUgsyxdihLsgeYU4WZk6xYfLfJAjo0FJQbJusaf7jsz2Yh6jQPtfd+F3Lni+1GWc9MQH3czJ8qtISkse0= 2023-11-06 23:25:48,089 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: EXPIRES=2023-11-06T23:40:47.341Z 2023-11-06 23:25:48,090 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: HOST=192.168.0.152 2023-11-06 23:25:48,090 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) HEADER: CONTENT_LENGTH=5372111111 2023-11-06 23:25:48,092 INFO [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) URI: uuid=67f72e90-a981-4d76-a7ea-03a619d1172d 2023-11-06 23:25:48,147 ERROR [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-1:null) post request validation failed com.cloud.exception.InvalidParameterValueException: signature validation failed.         at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.validatePostUploadRequest(NfsSecondaryStorageResource.java:3536)         at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:165)         at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelRead0(HttpUploadServerHandler.java:65)         at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)         at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)         at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:328)         at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:315)         at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:429)         at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)         at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)         at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)         at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)         at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)         at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)         at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)         at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)         at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)         at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)         at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)         at java.base/java.lang.Thread.run(Thread.java:829)

Upload progress dialog continues after the exception is thrown.

Ron

On 11/6/2023 6:14 PM, Wei Zhou wrote:

That is the first thing I tried when this started messing up (I
was able
to upload 3 ISOs before it stopped working).

Apparently Chrome has memorized the cert for the SSVM because
browsing
to the IP address - in my case: https://192.168.0.152 - does not
present
an "accept this cert" dialog.  In fact, if this was the problem,
then
wouldn't the upload fail to even start (I get the upload
progress dialog
without a problem).

I had found similar issues to this one on GitHub (#7442
<https://github.com/apache/cloudstack/issues/7442>, #7981
<https://github.com/apache/cloudstack/issues/7981>). The
resolution to that issue mentioned something about a firewall
rule to
the SSVM.  For grins, I tried that and no difference was
encountered (or
I misunderstood where to modify the firewall rule).

Any chance for other things to try?

Ron Gage

On 11/6/2023 3:20 AM, dahn wrote:

    Ah, this has to do with certificates. Uploading from local
    will fail
    this way. The trick is to try an

      * upload while having the debugger/network log open in
        your browser
      * get the actual upload url
      * open another window
      * go to the base url of the upload
      * accept the server cert
      * go back and try to upload

    let me know if this works @rongage
    <https://github.com/rongage> https://github.com/rongage

    —
    Reply to this email directly, view it on GitHub
    #8175 (comment)
    <https://github.com/apache/cloudstack/issues/8175#issuecomment-1794291710>,
    or unsubscribe
    https://github.com/notifications/unsubscribe-auth/ALT2YJCUMUAQVNQCHZXCPADYDCMWHAVCNFSM6AAAAAA62DAMISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJUGI4TCNZRGA.
    You are receiving this because you were mentioned.Message ID:
    /*@*/.***>

Can you check if there are errors in /var/log/cloud.log in SSVM?

— Reply to this email directly, view it on GitHub https://github.com/apache/cloudstack/issues/8175#issuecomment-1797012554, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALT2YJCJBY3YQMDMBUNYNELYDFVOLAVCNFSM6AAAAAA62DAMISVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJXGAYTENJVGQ. You are receiving this because you were mentioned.Message ID: @.***>

DaanHoogland commented 10 months ago

I had found similar issues to this one on GitHub (#7442, #7981). The

those issues are closed. #7442 in 4.18.1 As you report this in that version, How is this different? It seems identical (but I am a bad reader;)

DaanHoogland commented 8 months ago

@rongage is this still an issue?