Unexpected single newline character in chunk size

[misselvexu]

Expected Behaviour

(Please describe the behaviour you expected to see. If possible please refer to paragraphs of the official tus protocol specification on https://tus.io/.)

upload failed with exception

Actual Behaviour

(What faulty behaviour does the implementation have?)

java.io.IOException: Protocol violation: Unexpected single newline character in chunk size at me.desair.tus.server.util.HttpChunkedEncodingInputStream$ChunkSizeState$2.process(HttpChunkedEncodingInputStream.java:389) at me.desair.tus.server.util.HttpChunkedEncodingInputStream.readChunkSizeInformation(HttpChunkedEncodingInputStream.java:214) at me.desair.tus.server.util.HttpChunkedEncodingInputStream.getChunkSize(HttpChunkedEncodingInputStream.java:189) at me.desair.tus.server.util.HttpChunkedEncodingInputStream.nextChunk(HttpChunkedEncodingInputStream.java:164) at me.desair.tus.server.util.HttpChunkedEncodingInputStream.read(HttpChunkedEncodingInputStream.java:120) at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:99) at java.security.DigestInputStream.read(DigestInputStream.java:161) at java.security.DigestInputStream.read(DigestInputStream.java:161) at java.security.DigestInputStream.read(DigestInputStream.java:161) at java.security.DigestInputStream.read(DigestInputStream.java:161) at java.security.DigestInputStream.read(DigestInputStream.java:161) at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) at sun.nio.ch.FileChannelImpl.transferFromArbitraryChannel(FileChannelImpl.java:673) at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:711) at me.desair.tus.server.upload.disk.DiskStorageService.append(DiskStorageService.java:158) at me.desair.tus.server.core.CorePatchRequestHandler.process(CorePatchRequestHandler.java:49) at me.desair.tus.server.util.AbstractTusExtension.process(AbstractTusExtension.java:49) at me.desair.tus.server.TusFileUploadService.executeProcessingByFeatures(TusFileUploadService.java:293) at me.desair.tus.server.TusFileUploadService.processLockedRequest(TusFileUploadService.java:279) at me.desair.tus.server.TusFileUploadService.process(TusFileUploadService.java:172) at me.desair.tus.server.TusFileUploadService.process(TusFileUploadService.java:155) at com.acmedcare.tiffany.framework.nas.rest.server.NasServerBootstrap$FileUploadController.processUpload(NasServerBootstrap.java:112) at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877) at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:800) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1471) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745)

Steps to reproduce

(Ideally, you specify detailed steps using the curl command.)

Use tus-java-client Demo Code

// Create a new TusClient instance
TusClient client = new TusClient();

// Configure tus HTTP endpoint. This URL will be used for creating new uploads
// using the Creation extension
client.setUploadCreationURL(new URL("https://master.tus.io/files"));

// Enable resumable uploads by storing the upload URL in memory
client.enableResuming(new TusURLMemoryStore());

// Open a file using which we will then create a TusUpload. If you do not have
// a File object, you can manually construct a TusUpload using an InputStream.
// See the documentation for more information.
File file = new File("./cute_kitten.png");
final TusUpload upload = new TusUpload(file);

System.out.println("Starting upload...");

// We wrap our uploading code in the TusExecutor class which will automatically catch
// exceptions and issue retries with small delays between them and take fully
// advantage of tus' resumability to offer more reliability.
// This step is optional but highly recommended.
TusExecutor executor = new TusExecutor() {
    @Override
    protected void makeAttempt() throws ProtocolException, IOException {
        // First try to resume an upload. If that's not possible we will create a new
        // upload and get a TusUploader in return. This class is responsible for opening
        // a connection to the remote server and doing the uploading.
        TusUploader uploader = client.resumeOrCreateUpload(upload);

        // Alternatively, if your tus server does not support the Creation extension
        // and you obtained an upload URL from another service, you can instruct
        // tus-java-client to upload to a specific URL. Please note that this is usually
        // _not_ necessary and only if the tus server does not support the Creation
        // extension. The Vimeo API would be an example where this method is needed.
        // TusUploader uploader = client.beginOrResumeUploadFromURL(upload, new URL("https://tus.server.net/files/my_file"));

        // Upload the file in chunks of 1KB sizes.
        uploader.setChunkSize(1024);

        // Upload the file as long as data is available. Once the
        // file has been fully uploaded the method will return -1
        do {
            // Calculate the progress using the total size of the uploading file and
            // the current offset.
            long totalBytes = upload.getSize();
            long bytesUploaded = uploader.getOffset();
            double progress = (double) bytesUploaded / totalBytes * 100;

            System.out.printf("Upload at %06.2f%%.\n", progress);
        } while(uploader.uploadChunk() > -1);

        // Allow the HTTP connection to be closed and cleaned up
        uploader.finish();

        System.out.println("Upload finished.");
        System.out.format("Upload available at: %s", uploader.getUploadURL().toString());
    }
};
executor.makeAttempts();

[misselvexu]

Help Me~

[Maxoid]

Hi @misselvexu ! You are using Spring framework web container. So, the TUS service works with the request, which already decoded by this framework from the "chunks", but the Header "Transfer-Encoding"is still there and TUS service tryes to process them again (but there is no any chunks left after Spring web).

Just make a wrapper over the HttpRequest before invoking the TUS service, which omits returning "Transfer-Encoding" header. This should help

[tomdesair]

Hi @misselvexu,

I was able to reproduce your issue and it seems that @Maxoid is right (thx @Maxoid !). There is some double de-coding going on.

I've updated the code so that chunked HTTP decoding by the tus-java-server library is optional and disabled by default. Can you test if this solves your problem?

1. git clone https://github.com/tomdesair/tus-java-server.git
2. cd tus-java-server
3. mvn clean install
4. Change the dependency version of tus-java-server in the pom of your application to 1.0.0-1.1-SNAPSHOT
5. Rebuild your application and test again

If this fixes the issue, I'll release version 1.0.0-1.1 to Maven Central later this week.

[tomdesair]

I've did some more local testing and this issue is fixed in the new version which I've released now. Please update the dependency to:

<dependency>
    <groupId>me.desair.tus</groupId>
    <artifactId>tus-java-server</artifactId>
    <version>1.0.0-1.1</version>
</dependency>

If you still encounter this problem with the new version, feel free to reopen this issue.

[misselvexu]

Thanks @Maxoid & @tomdesair ; I haved fix this issues with suggest by @Maxoid ;

Thanks again;