File-upload-widget not working on ODK Aggregate v2.0.2

mmarciniak90 commented 5 years ago

Software and hardware versions

Aggregate v2.0.2 Problem is not visible on Aggregate 1.7

Problem description

Steps to reproduce the problem

User uploads files (.jpg, .txt, .doc, .jpeg, .csv, .mp3, .png, .gif) and sends them to Aggregate
Only .jpg files are available. Other submissions are not visible

fileWidgets.xml.txt

1.7	2.0.2

Expected behavior

No data lost

Other information

Reported on forum: https://forum.opendatakit.org/t/file-upload-widget-not-working-on-odk-aggregate-v2-0-2/19164

yanokwa commented 5 years ago

@ggalmazor Since this is a regression, I think we should try to get it into the next point release of Aggregate. Agreed?

ggalmazor commented 5 years ago

Thanks for reporting the issue! I've been able to reproduce this issue with v2.0.2 and I can confirm too that with v1.7.2 it doesn't happen.

In v1.7.2, the SubmissionServlet.doPost() method receives this POST payload:

--FjYdwmUYlbN1DPdhLTs1hGCscazCtBdI
Content-Disposition: form-data; name="xml_submission_file"; filename="file widget_2019-04-19_11-18-46.xml"
Content-Type: text/xml; charset=ISO-8859-1
Content-Transfer-Encoding: binary

<?xml version='1.0' ?><file-widget id="file-widget" xmlns:h="http://www.w3.org/1999/xhtml" xmlns:jr="http://openrosa.org/javarosa" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:orx="http://openrosa.org/xforms" xmlns:odk="http://www.opendatakit.org/xforms"><file>1555665534497.txt</file><meta><instanceID>uuid:df890be0-7a61-43ff-9285-6d13486cc015</instanceID></meta></file-widget>
--FjYdwmUYlbN1DPdhLTs1hGCscazCtBdI
Content-Disposition: form-data; name="1555665534497.txt"; filename="1555665534497.txt"
Content-Type: text/plain
Content-Transfer-Encoding: binary

Hola

--FjYdwmUYlbN1DPdhLTs1hGCscazCtBdI--

Then, when I try to send the same submission to a v2.0.2 server, I get this POST payload:

--ADKi8EuQJXh7CMrysMHL6175mKCrgsouqOu8
Content-Disposition: form-data; name="xml_submission_file"; filename="file widget_2019-04-19_11-18-46.xml"
Content-Type: text/xml; charset=ISO-8859-1
Content-Transfer-Encoding: binary

<?xml version='1.0' ?><file-widget id="file-widget" xmlns:h="http://www.w3.org/1999/xhtml" xmlns:jr="http://openrosa.org/javarosa" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:orx="http://openrosa.org/xforms" xmlns:odk="http://www.opendatakit.org/xforms"><file>1555665534497.txt</file><meta><instanceID>uuid:df890be0-7a61-43ff-9285-6d13486cc015</instanceID></meta></file-widget>
--ADKi8EuQJXh7CMrysMHL6175mKCrgsouqOu8--

Note that the second part of the multipart payload with the txt file attachment is missing now.

This shows that Aggregate is rejecting some parts of the message before even getting to the servlet, which indicates some web configuration issue.

I Will continue investigating this issue.

ggalmazor commented 5 years ago

After being unable to see any meaningful difference in the webapp's configuration between v1.7.2 and v2.0.2, I've verified with ettercap (sniffer) that Collect doesn't send the part that we're missing. This could have to do with the pre-flight HEAD request it does before sending the submission. I'll continue studying that.

ggalmazor commented 5 years ago

I've found that the responses to the pre-flight HEAD requests Collect makes are different:

v1.7.2

HTTP/1.1 204 No Content.
Date: vie, 19 abr 2019 10:52:37 GMT.
X-OpenRosa-Version: 1.0.
X-OpenRosa-Accept-Content-Length: 10485760.
Location: http://192.168.1.129:8080/submission.
Server: Jetty(9.2.22.v20170606).

v2.0.2

HTTP/1.1 204 No Content.
Date: vie, 19 abr 2019 10:40:02 GMT.
X-OpenRosa-Version: 1.0.
X-OpenRosa-Accept-Content-Length: 10485760.
Server: Jetty(9.2.22.v20170606).

This one is missing the Location header. We removed that header because HTTP 204 responses should not have it

ggalmazor commented 5 years ago

I've verified that after restoring the Location header to the pre-flight HEAD response, then Collect sends all the parts, fixing the issue.

I think we need to discuss this over at Collect, because it seems weird that it wouldn't send a complete request depending on the presence of that header, more so if we aknowledge that the header is not giving Collect any valuable information for sending submissions and that HTTP 204 responses shouldn't have it in the first place.

What do you think, @grzesiek2010?

issa-tseng commented 5 years ago

investigating this report, i think Central is suffering from the same problem. as far as i can tell, we have always suffered from this issue in Central, because we've never supplied this Location header.

lognaturel commented 5 years ago

See https://github.com/opendatakit/collect/pull/3038 for a proposed fix.

getodk / aggregate