Error receiving from chunked transfer encoding

[imvis]

Hello @phax,

Your library works amazing for regular AS2 messages but I'm having trouble receiving chunked encoded messages, it keeps throwing an error "The value of 'Offset' must be >= 0! The current value is: -1".

The method in question is com.helger.as2lib.util.http.TempSharedFileInputStream.storeContentToTempFile(InputStream, String) at line 143 when it's trying to store the inputstream into a temporary file using com.helger.commons.io.stream.StreamHelper.CopyByteStreamBuilder.build().

I'm using BizTalk 2016 to send the AS2 message with a 23KB text file over to my JAVA program with the below configuration:

The error messages that were printed in the console are:

2022-06-29 16:22:10.163  INFO 27852 --- [nio-8080-exec-8] .a.AbstractAS2ReceiveBaseXServletHandler : Starting to handle incoming AS2 request - 192.168.50.100:50178
2022-06-29 16:22:10.687  INFO 27852 --- [nio-8080-exec-8] c.helger.as2lib.exception.AS2Exception   : as2-lib 4.10.1 IllegalArgumentException terminated: Failed to read Servlet Request: The value of 'Offset' must be >= 0! The current value is: -1
2022-06-29 16:22:10.687 ERROR 27852 --- [nio-8080-exec-8] .a.AbstractAS2ReceiveBaseXServletHandler : Not having a data source to operate on

The as2-servlet version I'm using is:

<dependency>
    <groupId>com.helger.as2</groupId>
    <artifactId>as2-servlet</artifactId>
    <version>4.10.1</version>
</dependency>

Thank you for maintaining the library, would really appreciate it if you can shed some light as to whether the behavior is intended or is there any way to resolve this.

[imvis]

2022-06-29-14-47-02-1409.txt

This is an example of the file I'm using as a test if you need something to try out with. I further tested with some variations of file contents and it seems like the first line of the file is causing the problem, even something as simple as "12345" will result in the offset error. Without the first line in the above-attached text file, it seems like the AS2 message can be received successfully even with chunked encoding enabled for messages less than 10KB. For messages more than 10KB, although there wasn't any error triggered, the received file will become empty.

The combination I tried that consistently gets the error is when chunked encoding is enabled, the file size is greater than 10KB and the first line of the text file contains 5 characters or more.

[phax]

Hi, unfortunately I cannot find the exact source of the exception.

Instead I am pointing you to the source of HTTP chunked encoding: RFC 7230, section 4.1: https://datatracker.ietf.org/doc/html/rfc7230#section-4.1

According to the spec, each chunked block needs to start with the length of the block. This is missing in your example payload.

Correct input looks e.g. like this:

2
ab
1
c
0

[imvis]

Thanks for your response, it seems to be working for me now after commenting out lines 236 to 240 in HTTPHelper.java, importing the Apache Common IO library, and adding this line of code below the commented lines:

aBytePayload = IOUtils.toByteArray(aIS);

It seems like the input stream at this point in time already has the complete message and all I had to do is convert it to a byte array and pass it through to the rest of the functions. Do you know if there would be any implication in changing the codes this way?

[phax]

Well, the error is clearly on the sender side. If they claim to use "chunked encoding" they need to follow the specification. Outcommenting these lines means handling the messages as a "non-chunked" (hence a "regular") transmission.

With this commenting, you will not be able to received chunked encoding messages from any sender that is using it correctly.

[phax]

P.S. somebody wrote an article on that: "BizTalk AS2 and Chunked Encoding (Disable it!)" https://jason.agostoni.net/2010/09/01/biztalk-as2-and-chunked-encoding-disable-it/

[imvis]

That's rather unexpected from a Microsoft product but I guess it's not the only middleware that behaves like this, ArcESB as well. Do you think it's possible to add a check if the decoding of a chunked message fails, you failover to use the input stream as a regular transmission?

[phax]

Nope, that won't work. Because if chunked encoding reading fails in the middle, then the previous parts of the InputStream have already been consumed. The only potential way would be to always store the data in a temporary file and read from there. But that would lead to severe performance degradations etc.

[imvis]

Seems like I misjudged Biztalk, I installed Wireshark on the server my Biztalk was installed in to monitor the traffic being sent to the JAVA app and noticed that the request is indeed chunked:

I went on to look at the filter chains while in debug mode, I noticed that during the preprocessing of the request at Http11Processor, it encapsulated ChunkedInputFilter to the stream and that might be the culprit doing the parsing of the chunked data when the input stream is being read hence I'm able to transform it into the byte array and use it like that.

Not sure if anyone else's instance of the library behaves like this, I tested with my own JAVA app utilizing the as2-servlet library as well as the as2-demo-webapp from the repository and both resulted in the same behavior. I'm running these JAVA apps using Tomcat so this might be something specific to that.

[phax]

Okay, I will look at it again - thanks for the deep-dive :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[phax]

Version 5.1.1 contains a fixed for chunked encoding issue