Intermittent issues with pushing surveys with attached photos

[KCI-Ablowers]

@talllguy

We had no issues with taking photos in the survey and pushing them either day we were in the field last week (May 2nd & 3rd). The app worked like a dream. In the office on Friday, I started to push the paper surveys we did on 04/13/2018 into production via Survey123 with attached photos from the iPad's storage. I had no issues with my first two surveys being pushed, even with a number of photos (SWMFAC #100188 and #100198). The third survey (SWMFAC #100133) failed multiple times when sending attachments. It would send the first few images, then fail. A record was still created in the feature service along with records in the R_PHOTOS table, but not all of the images would be attached. The work around that seemed to solve it was to remove the photos, submit the survey, copy the sent survey to a new form, delete the inspection from the feature service on AGOL, reattach the photos to the copied survey, and send. This worked for 2 surveys (SWMFAC #100133 and #130131) and all other images were viewable on AGOL. The 5th survey (SWMFAC #130431) would not push at all, with any of the images, even after following the "copy, delete and send" method described above. I was finally able to submit it this morning by testing which photos would actually push.

Does the survey itself limit the size of photo attachments? can this limit be removed (if it is applied currently). Or, is this a features service issue? The attachment issue came up previously but seemed to be resolved. We had no issues with taking taking and sending photos while in the survey form.

Brent and I are beginning to think taking the photos on the iPad's camera, attaching, and submitting may be the better method moving forward.As long as the user is aware what questions require a photo for 3,4, or 5 ratings and can remember which photos are which. To do this though, we need to iron out the attachment issue.

[johnshiu]

Since it sounds like you were able to get some working by recreating and attaching the same photos as before, I don't think it's a file size issue. I had fixed a file size issue earlier relating to IIS, but that should've fixed it I think (email excerpt below for reference):

I’ve been debugging this issue, and I believe I have may have fixed the problem. Looking at the error being thrown, it seems to be causing an HTTP 413 error, meaning that the server is rejecting the request due to size. This makes sense why some photos may work and others don’t; smaller, simpler ones like a black test screen would work since they are small, while real photos with lots of detail likely wouldn’t.

Anyway, I increased the max request size value in IIS on the DMZ server to 50 MB, and I tried uploading a rather large photo (~5 MB) via a survey in Chrome, and it appears to have worked. Please test it out and let me know if it works on your end.

[KCI-Ablowers]

I think this issue is/was resolved on your end, @johnshiu. I forgot it pertained to the IIS and not anything AGOL or Survey123 related.

[KCI-Ablowers]

Submitted an inspection for SWMFAC # 132101 with 11 attached photos from the iPad with no issues. Thinking the issue may be down to the photos themselves that I was trying to push, which if true, does not bode well for the inspection workflow of "Photos first, Survey second".

[talllguy]

double check the photo size is set.

[talllguy]

Confirmed max image width is set to 1280 pixels, so there shouldn't be any issues saving.

See https://github.com/kcigeospatial/MDOT-SHA-NPDES-Next-Gen/blob/master/config/survey123/NNG_BMPInspection_info.info#L28-L29

[KCI-Ablowers]

Had an issue with submitting photos from a regular digital camera. The photos came off the camera at 4000x3000 pixels. The first survey I submitted only had one image which was left at its original 4000x3000 pixel resolution. There was no issue pushing this survey. This was SWMFAC #022263.

The second survey had five image attachments at 4000x3000 pixels and failed to push images 2-5. I reduced the resolution to 1280x960 using GIMP and reattached to resubmit. Images 3-5 failed to push this time. Further reduced the resolution to 1080x810 and all photos pushed with no issue. This was SWMFAC #022264.

@talllguy or @johnshiu is there a maximum attachment size generated on the fly by Survey123, the feature service, something on the machine hosting the service, etc.? It seems odd it would take one large photo, and then only two at a smaller resolution, then five at an even smaller resolution. We had this issue previously with photos attached from the iPad. Should I add a section to my SOP(Data Management) doc explaining that attaching photos can cause errors and you may need to reduce the resolution for them to push?

[johnshiu]

Can you send me the originals of the photos for the one that worked and the ones that didn’t? I think it could be a file size issue most likely.

[johnshiu]

@KCI-Ablowers

Looking at these, based on file size, the first was 4 MB. The remaining 5 are 25.5 MB.

I wonder if there is an issue with image sets over a certain size. I think I will try testing it using the KCI_1 version, which I will delete later.

[KCI-Ablowers]

Can you hold off on testing? I'm waiting for @impittman to push my 3 inspections through to production. Hopefully she can do it sometime today.

For the photo size, I am thinking we may have to scrap the attachments all together. It adds too much to the workflow (in my opinion). @talllguy any thoughts?

[KCI-Ablowers]

@johnshiu Unless you are only talking about editing the KCI_1 field feature service then disregard my previous comment.

[johnshiu]

@KCI-Ablowers Yes, I was just going to use the KCI_1 feature service and then delete them later.

[johnshiu]

@KCI-Ablowers Just to confirm, when you uploaded these, was it through the desktop S123 app on existing surveys?

[KCI-Ablowers]

@johnshiu The survey was initially created on the S123 desktop application. When it failed, I editing the existing survey(what I attempted to submit) using the "Do you want to edit this existing survey?" option that shows in the outbox.

[talllguy]

@KCI-Ablowers do you happen to know what version of desktop you were using? A similar issue was just fixed according to the update channel.

[talllguy]

@johnshiu Andrew and I tested this extensively today with an inspection he did in the field today with 13 photos that were taken in the survey form. It repeatedly fails after uploading the 8th photo. I ran the console logger to capture what happened. We see a HTTP 500 error right when it fails uploading. Can you investigate on the server at the timestamp in this log file to see what is happening?

030516.txt

[johnshiu]

Looking at the server logs, I see the 7 successful attachment adds; the 8th doesn't show up in the GISMOB1 logs at all:

Also, in the ArcGIS server log on DMZWEB1, I found this attachment error:

Failed to add attachment. File size limit of 2,000 MB exceeded for feature service '/arcgis/rest/services/NPDES_Next_Gen/NNG_BMP_Inspection_Prod_20180424/FeatureServer'.

The time doesn't match up, however, so I think that's a different problem.

[johnshiu]

Tried the following changes; on IIS on GISMOB1, set the arcgis request filtering options as follows (maxAllowedContentLength, maxQueryString, maxUrl):

Also, I increased the serverRuntime uploadReadAheadSize to 1 GB:

Can you please try to upload the photos again?

[talllguy]

I made a test inspection with 20 photos of The Office playing on TV and it appears to fail after just two of them. Just submitted, around 8:02pm

[talllguy]

Ran it again with the AppStudio console running. It failed at the same place, just 2 photos in.

[johnshiu]

Tried adding my own survey and attachments via the desktop app, and it failed on the second photo. Also gives me a useless error message: HTTP ERROR 500

I can't find anything yet in the server logs that tells me the root cause of this error, but I'm going to keep looking.

[talllguy]

Check if it has anything to do with the output folder? I have seen this in multiple places as a possibility https://community.esri.com/thread/57654

[johnshiu]

Hmm, I don't think so, since I would think if the output folder is not mapped properly, then none of the map services would be able to serve out tiles/map exports.

[johnshiu]

One thing I'd like to try is to create a survey using the NPDES service URL without the proxy through AGOL. It's possible that the AGOL proxy is blocking the request, which may explain why we are not getting any log information (errors or not) for subsequent uploads.

@talllguy, do you think you'd be able to set that up for testing somehow?

[talllguy]

@johnshiu I will attempt this. The trick is that the submit URL is generated based on the proxied service URL. Currently that URL is:

https://www.arcgis.com/sharing/rest/content/items/273a22bacf514ee7beb9abb3a446efa3

How can we generate a similar URL for this feature service directly?

[talllguy]

First test, just dropping the URL in there as is

https://maps.roads.maryland.gov/arcgis/rest/services/NPDES_Next_Gen/NNG_BMP_Inspection_Prod_20180424/FeatureServer

That failed. Maybe I can generate a token for the security.

[talllguy]

Maybe this will work

[johnshiu]

Hmm, I see what you mean; it may not be possible to directly access this.

Is there any other way we can try to add attachments? Perhaps not through Survey123 but through AGOL/ArcMap/ArcGIS Pro? Based on the logs, it looks like the request doesn't even make it through to the GISMOB1/DMZWEB1 server, which is very odd.

[talllguy]

We can upload attachments in the web editor. Shall I try attaching a large file there and seeing if it balks?

[johnshiu]

Yes, please try uploading through there and see if it ever fails.

[talllguy]

Will do. Also, according to this doc, the feature service must be added to AGOL via the Proxy (as we've been doing): https://support.esri.com/en/technical-article/000014793

[talllguy]

OK, it is also failing through the web. The same nondescript 500 error. Here's the test process:

[talllguy]

Here's that error log. https://gist.github.com/talllguy/9a8a2972e504dd1cd920b8546d6c803c

[johnshiu]

Thanks for testing, Elliott. I'll investigate this further.

[talllguy]

John, I couldn't resist doing a little more testing. I was able to successfully add a large attachment using the REST API addAttachment directly. Here are the steps:

1. go to https://maps.roads.maryland.gov/arcgis/rest/services/NPDES_Next_Gen/
2. log in
3. https://maps.roads.maryland.gov/arcgis/rest/services/NPDES_Next_Gen/NNG_BMP_Inspection_Prod_20180424/FeatureServer/16/3223
4. click addAttachment
5. find a suitable file
6. upload
7. 💚 success
8. check attachment success on REST API

Here's a video of me validating and then adding a second attachment to feature 3223 in the Photos table. You can see that the size is 12.2 megapixels, so the feature service itself has no problem with the large files.

[johnshiu]

I've been testing as well, and I've found that certain files just won't get pushed through. Of the 6 sample photos Andrew sent me, 4 always work fine, and 2 always fail:

This is pretty crazy behavior. Also, the 2 that fail aren't necessarily larger than the ones that work.

[johnshiu]

Based on your tests, it sounds like the proxy is doing something funny and rejecting certain files. Let me try the _error files directly using the REST endpoint like you did.

[johnshiu]

Hmm, the _error files seem to work perfectly. There is definitely something up with the AGOL proxy. I will continue to test, this time against the ECO environment. If we still encounter errors, we may have to take it up with ESRI.

[johnshiu]

Okay, so I tried using the ECO server, and like at SHA, the upload fails through AGOL but works when used directly against the REST interface.

I think the last thing to test is whether a fully-hosted survey will permit these attachments to be uploaded. I suspect a hosted one might work if it's not using the same programming logic as the proxy.

[talllguy]

I can stand up a copy of the form as a hosted service on AGOL. Do you want to test if the particular ones that fail (_error) fail there too?

[johnshiu]

AGOL_photo_attachment_errors.zip

Here are the two sample photos that consistently fail.

[talllguy]

Thanks. Those files appear normal and open alright in windows.

I copied the form and published as hosted. Test 1, the 1587 file. Result: pass. It uploaded, no problem.

This is looking more and more like a proxy issue.

[johnshiu]

Thanks for testing the hosted version. Yes, it definitely seems like a proxy issue. The good news I suppose is that we have test files that reliably fail, so we could provide it to ESRI. Very curious what causes it to throw a 500 error.

[talllguy]

This seems like a good indicator of the problem, something to do with how HTTP/1.1 handles chunking. We can reference the service ticket when contacting Esri. https://community.esri.com/thread/196673-what-could-be-causing-attachments-to-fail-in-a-geoform-submission

[johnshiu]

That's an interesting find. In our case, we don't get the actual CORS error, although the symptoms seem very similar.

Also, I don't think the user's proposed solution of using a custom proxy would work in our case, since we need to use AGOL for user authentication. Also, I think we had failures with smaller files, too; not just those that were over a certain size. I was able to upload several 5 MB photos, but only some were repeatably failing to get pushed through.

[talllguy]

Esri Case initiated! 02127148 is our case ID. Craig and I just initiated it.

[talllguy]

@johnshiu does this bug sound applicable to you?

BUG-000107195: ArcGIS Online Utility Proxy will fail to include an access-control-allow-origin header when attaching certain files to secure feature services which have been added to ArcGIS Online with stored credentials.

[johnshiu]

Yes, this bug could be applicable. However, we do not get the CORS error (access-control-allow-origin error) that the other user encountered in his thread.

[talllguy]

I did not notice the CORS issue when looking at the Chrome logs.

[talllguy]

I referenced their case ID in the Esri case. The timeliness of that last comment with the bug is amazing. Just came in today.

[talllguy]

I tried putting port 6443 in the URL but it doesn't resolve. Perhaps our setup uses another port?