vaadin / flow

Vaadin Flow is a Java framework binding Vaadin web components to Java. This is part of Vaadin 10+.
Apache License 2.0
616 stars 167 forks source link

Multi File Upload doesn't work #17705

Open scrmtrey91 opened 1 year ago

scrmtrey91 commented 1 year ago

Description

We have form that user can upload files to a document.

If we use MemoryBuffer and only 1 file upload max, it is working no problem.

When we use MultiFileMemoryBuffer and multiple files upload at once, then the we encounter a problem.

Blocking read timeout java.io.IOException: Blocking read timeout
    at org.glassfish.grizzly.http2.DefaultInputBuffer.poll0(DefaultInputBuffer.java:271) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http2.DefaultInputBuffer.poll(DefaultInputBuffer.java:236) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http2.Http2Stream.pollInputData(Http2Stream.java:663) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http2.Http2BaseFilter.checkIfHttp2StreamChain(Http2BaseFilter.java:792) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http2.Http2ServerFilter.handleRead(Http2ServerFilter.java:428) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.read(DefaultFilterChain.java:328) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.filterchain.FilterChainContext.read(FilterChainContext.java:713) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http.io.InputBuffer.blockingRead(InputBuffer.java:1113) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http.server.io.ServerInputBuffer.blockingRead(ServerInputBuffer.java:72) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http.io.InputBuffer.fill(InputBuffer.java:1137) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http.io.InputBuffer.read(InputBuffer.java:334) ~[nucleus-grizzly-all.jar:?]
    at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:231) ~[web-core.jar:?]
    at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:260) ~[web-core.jar:?]
    at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:1027) ~[commons-fileupload-1.5.jar:1.5]
    at org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:931) ~[commons-fileupload-1.5.jar:1.5]
    at java.io.InputStream.read(InputStream.java:218) ~[?:?]
    at com.vaadin.flow.server.communication.StreamReceiverHandler.streamToReceiver(StreamReceiverHandler.java:536) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.communication.StreamReceiverHandler.handleFileUploadValidationAndData(StreamReceiverHandler.java:412) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.communication.StreamReceiverHandler.handleStream(StreamReceiverHandler.java:316) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.communication.StreamReceiverHandler.handleMultipartFileUploadFromInputStream(StreamReceiverHandler.java:270) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.communication.StreamReceiverHandler.doHandleMultipartFileUpload(StreamReceiverHandler.java:202) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.communication.StreamReceiverHandler.handleRequest(StreamReceiverHandler.java:159) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.communication.StreamRequestHandler.handleRequest(StreamRequestHandler.java:116) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.VaadinService.handleRequest(VaadinService.java:1560) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.VaadinServlet.service(VaadinServlet.java:369) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.cdi.CdiVaadinServlet.service(CdiVaadinServlet.java:67) ~[vaadin-cdi-14.1.1.jar:?]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:584) ~[jakarta.servlet-api.jar:4.0.4]
    at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1637) ~[web-core.jar:?]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:331) ~[web-core.jar:?]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211) ~[web-core.jar:?]
    at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:282) ~[tyrus-container-servlet.jar:?]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:253) ~[web-core.jar:?]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211) ~[web-core.jar:?]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:257) ~[web-core.jar:?]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:167) ~[web-core.jar:?]
    at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:757) ~[web-core.jar:?]
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577) ~[web-core.jar:?]
    at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) ~[web-glue.jar:?]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:158) ~[web-core.jar:?]
    at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:372) ~[web-core.jar:?]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:239) ~[web-core.jar:?]
    at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520) ~[kernel.jar:?]
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217) ~[kernel.jar:?]
    at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) ~[nucleus-grizzly-all.jar:?]
    at java.lang.Thread.run(Thread.java:833) ~[?:?]
]]

Sometimes it doesnt rise immediatly but lets say after 10-15s (even if user close the form). So probably something broken in the background of MultiFileMemoryBuffer (probably with thread safety related).

We also use automatic PUSH websocket xhr.

The problem occur in dev and production.

Also the other problem is that application after this upload error the whole server is unusable, all web services, UI throw "read timeout exceptions", all stop working.

[2023-09-20T09:42:47.645+0200] [Payara 5.2022.5] [INFO] [] [] [tid: _ThreadID=4027 _ThreadName=pool-38-thread-93] [timeMillis: 1695195767645] [levelValue: 800] [[
  [pool-38-thread-93] 09:42:47.644 INFO  com.vaadin.flow.server.communication.AtmospherePushConnection 
 Error waiting for messages to be sent to client before disconnect java.lang.InterruptedException: null
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081) ~[?:?]
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:276) ~[?:?]
    at org.atmosphere.cpr.BroadcasterFuture.get(BroadcasterFuture.java:125) ~[atmosphere-runtime-2.7.3.slf4jvaadin4.jar:?]
    at com.vaadin.flow.server.communication.AtmospherePushConnection.disconnect(AtmospherePushConnection.java:355) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.component.internal.UIInternals.setPushConnection(UIInternals.java:472) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.component.internal.UIInternals.setSession(UIInternals.java:424) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.VaadinSession.removeUI(VaadinSession.java:621) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.VaadinService.lambda$fireSessionDestroy$20ed7015$1(VaadinService.java:648) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.component.UI.accessSynchronously(UI.java:466) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.component.UI.accessSynchronously(UI.java:427) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.VaadinService.lambda$fireSessionDestroy$9c853e43$1(VaadinService.java:638) ~[flow-server-23.3.18.jar:23.3.18]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
    at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) ~[?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java) ~[?:?]
    at com.vaadin.flow.server.VaadinService.runPendingAccessTasks(VaadinService.java:2055) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.VaadinSession.unlock(VaadinSession.java:712) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.VaadinService.ensureAccessQueuePurged(VaadinService.java:2021) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.VaadinService.accessSession(VaadinService.java:1988) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.VaadinSession.access(VaadinSession.java:1016) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.component.UI.access(UI.java:538) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.component.UI.access(UI.java:521) ~[flow-server-23.3.18.jar:23.3.18]`

`[2023-09-20T09:42:37.660+0200] [Payara 5.2022.5] [SEVERE] [] [] [tid: _ThreadID=57 _ThreadName=http-thread-pool::http-listener-2(5)] [timeMillis: 1695195757660] [levelValue: 1000] [[
 IO Exception during file upload, fired as StreamingErrorEvent java.io.IOException: Blocking read timeout
    at org.glassfish.grizzly.http2.DefaultInputBuffer.poll0(DefaultInputBuffer.java:271) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http2.DefaultInputBuffer.poll(DefaultInputBuffer.java:236) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http2.Http2Stream.pollInputData(Http2Stream.java:663) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http2.Http2BaseFilter.checkIfHttp2StreamChain(Http2BaseFilter.java:792) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http2.Http2ServerFilter.handleRead(Http2ServerFilter.java:428) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.filterchain.DefaultFilterChain.read(DefaultFilterChain.java:328) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.filterchain.FilterChainContext.read(FilterChainContext.java:713) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http.io.InputBuffer.blockingRead(InputBuffer.java:1113) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http.server.io.ServerInputBuffer.blockingRead(ServerInputBuffer.java:72) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http.io.InputBuffer.fill(InputBuffer.java:1137) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.http.io.InputBuffer.read(InputBuffer.java:334) ~[nucleus-grizzly-all.jar:?]
    at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:231) ~[web-core.jar:?]
    at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:260) ~[web-core.jar:?]
    at org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:1027) ~[commons-fileupload-1.5.jar:1.5]
    at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:971) ~[commons-fileupload-1.5.jar:1.5]
    at org.apache.commons.fileupload.MultipartStream$ItemInputStream.close(MultipartStream.java:950) ~[commons-fileupload-1.5.jar:1.5]
    at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl$FileItemStreamImpl.close(FileUploadBase.java:903) ~[commons-fileupload-1.5.jar:1.5]
    at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.findNextItem(FileUploadBase.java:1061) ~[commons-fileupload-1.5.jar:1.5]
    at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.hasNext(FileUploadBase.java:1151) ~[commons-fileupload-1.5.jar:1.5]
    at com.vaadin.flow.server.communication.StreamReceiverHandler.handleMultipartFileUploadFromInputStream(StreamReceiverHandler.java:268) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.communication.StreamReceiverHandler.doHandleMultipartFileUpload(StreamReceiverHandler.java:202) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.communication.StreamReceiverHandler.handleRequest(StreamReceiverHandler.java:159) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.communication.StreamRequestHandler.handleRequest(StreamRequestHandler.java:116) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.VaadinService.handleRequest(VaadinService.java:1560) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.flow.server.VaadinServlet.service(VaadinServlet.java:369) ~[flow-server-23.3.18.jar:23.3.18]
    at com.vaadin.cdi.CdiVaadinServlet.service(CdiVaadinServlet.java:67) ~[vaadin-cdi-14.1.1.jar:?]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:584) ~[jakarta.servlet-api.jar:4.0.4]
    at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1637) ~[web-core.jar:?]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:331) ~[web-core.jar:?]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211) ~[web-core.jar:?]
    at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:282) ~[tyrus-container-servlet.jar:?]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:253) ~[web-core.jar:?]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211) ~[web-core.jar:?]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:257) ~[web-core.jar:?]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:167) ~[web-core.jar:?]
    at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:757) ~[web-core.jar:?]
    at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577) ~[web-core.jar:?]
    at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99) ~[web-glue.jar:?]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:158) ~[web-core.jar:?]
    at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:372) ~[web-core.jar:?]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:239) ~[web-core.jar:?]
    at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520) ~[kernel.jar:?]
    at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217) ~[kernel.jar:?]
    at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) ~[nucleus-grizzly-all.jar:?]
    at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) ~[nucleus-grizzly-all.jar:?]
    at java.lang.Thread.run(Thread.java:833) ~[?:?]
]]

We also pay for PRO subscription which is not cheap 130 EUR per month for broken components.

Expected outcome

Multi file upload should work out of the box with no errors and broken the whole server.

Minimal reproducible example

Empty app with push and upload component with no listeners.

Steps to reproduce

1) Deploy app 2) Mutli file upload (mix of pdf, png etc)... 3) Sometimes works, 90% time is broken

Environment

Vaadin version(s): 23 (latets, but present in early realeases also) Browser: Chrome latest / Edge latest Server: Payara 5.2022.5 JDK: Java 17 latest

Browsers

Chrome, Edge

mcentrih commented 1 year ago

I have the same problem in my app...

ugur-vaadin commented 1 year ago

Transferred the issue to Flow. It might also be related to Payara, there are multiple similar issues reported online.

scrmtrey91 commented 1 year ago

It looks like it is the problem with payara server. The same problem occurs on payara 6 aswell.

We tried wildfly for testing purposes and multi upload works no problem.

Is this fixable for payara server ?

We might be forced to use wildfly in the future because more and more problems with payara server.

knoobie commented 1 year ago

If you can already pin point the problem to payara, why aren't you opening an issue there as well?

scrmtrey91 commented 1 year ago

We didnt think it is payara issue until mentioned here.

We cant just change server in production thats why we didnt think is payara fault and that is another reason why we did not check with wildfly for example.

i did not even imagine that server can impac an UI component from vaadin.

scrmtrey91 commented 11 months ago

So i did a little reasearch.

And here is the "solution".

In payara i disable HTTP2 and only use HTTP1. Multi upload started working.

Also download file with special charachters start working again with http2 disabled.

Sometimes resources does not load (icon png images on dashboard in our app) Disable http2 and boom all good..

So it looks like combination payara + vaadin23 + http2 = bad experience

We also check vaadin 8 application and vaadin 8 also use http2 but we did not have problems like this.

caalador commented 11 months ago

Trying to debug this issue it seems that when we are in http2 the issue is that instead of the upload requests coming in in parts everything comes up at once for all files overwhelming the server which the blocks and timeouts. A small help was making the HTTP/2 Max Concurrent Streams: larger from the default 100 to for instance 1000, but much bigger and it didn't make a difference.

A better, but not complete fix was to add a sleep to upload when adding multiple files at once so anything calling addFile(file) having a small timeout for each file e.g. (file, i) => { setTimeout(()=>{this._addFile(file);},i*50); } Which basically fixed the issue when running on http2, but still fails on the blocking function when pushing big files (40+ megs)

Framework 8 didn't have the same style of multifile upload so thats probably why it didn't have the same issue.

caalador commented 11 months ago

This is to part the same problem that the client opens too many connections (as in vaadin/web-components#6698) due to the possibility of multiplexing in HTTP/2 sending all request concurrently.

mstahv commented 9 months ago

I would claim that this is a bug in our Upload component, not in Payara or Flow(framework part). It is by no means sane to fire unlimited amount of XHRs to upload each file separately. Either it should queue all files in one multipart requests or throttle the amount of concurrent connections as suggested by @caalador.

I was recently fiddling with other (only slightly related) upload issues and renewed the UploadFileHandler component in Viritin quite throughly. That is essentially Upload with better Java API. That component now (since version 2.5.1), in addition to making the API much simpler, limits the amount of concurrent uploads to one by default. Give it a shot and let me know if it don't work!