Problem with moth-trap uploads for long-lists?

[DavidRoy]

Related to: https://www.brc.ac.uk/irecord/node/8053#comment-5568

Suspect this relates to long-lists with multiple images.

I've found a list for me doesn't upload. Have just tried now (10:24am on 20th July 2020) if you can find anything in the logs. When clicking on upload there is a spinning wheel but the list does not move from pending to uploaded.

[kazlauskis]

Thanks, we didn't have the time to work on the iRecord App last week but we'll pick this up asap.

[kazlauskis]

@DavidRoy I have spent some time looking into this but the error message I can see gives me too little information. I have turned on a more verbose logging on the iRecord site (@BirenRathod - that's for a few days only) so if you could try to reupload the same record and let me know of the time then that would be great.

[DavidRoy]

Tried again at 18.30. We’ve had further reports of this issue

[kazlauskis]

Thanks, I have figured it out now. There are 2 issues that could be fixed within the website:

1. There is a limit of max 10 photo upload set on the website (maybe @BirenRathod knows more about it). This triggers 400 Missing media response here https://github.com/Indicia-Team/drupal-7-module-indicia-api/blob/24ca6861c975ebf4732adde1102b0e72055a5619/v1/samples/create.php#L464

2. For large uploads (eg. 100s of images or a record with very high quality images) a 413 Request Entity Too Large is returned. I am not sure what's the limit here but this might need increasing if we expect large surveys submitted.

Using a single submit request to send all the photos in one go might be good enough for small uploads but, ideally, once the data volume grows this should be improved and images should be allowed to be uploaded individually.

[BirenRathod]

@DavidRoy & @kazlauskis At present there is no limit of number of photo upload but there is a limit of 10MB. So total size of photo upload should not be exceed above than that. This limit apply to iRecord website as well.

Increasing limit is not a problem but there are some downsides to consider. By Allowing large number of files or size to upload, it will be compromised with network, so users need a good internet connection and good size data on mobile network. The other slightly less concern nowadays because of there are multiple firewalls in place but high risk in terms of security which is to avoid people to upload malware files on the server.

[kazlauskis]

I have tried to upload tiny set of images (less than 3mb in total) and also large ones (9mb in total), it looks like it didn't make the difference when it came to the 10 image limit - both couldn't be uploaded when the number exceeded 10.

[BirenRathod]

@kazlauskis, @DavidRoy & @johnvanbreda

That could be from Indicia code definitely not from PHP. I have added John to comment here.

[johnvanbreda]

I don't think this can be core Indicia code - the request is sent from the app to the Indicia API and the error occurs there, it doesn't reach the warehouse or Indicia iForm module. I can't see anything in the Indicia API which would limit the number of media files (though admittedly it's not my area of code expertise so I might have missed something).

@BirenRathod have you checked the PHP.ini max_file_uploads setting?

[BirenRathod]

@johnvanbreda Yes, I have it is 20MB on warehouse actually and 10MB on irecord website as I mentioned above.

[BirenRathod]

@kazlauskis Is there a default limit on iRecord app somewhere?

[kazlauskis]

There is only a timeout limit in the app but I am using a standalone tool (Postman) when testing the service here so this isn't something client related.

[johnvanbreda]

@BirenRathod no, not the max file size, the max_file_uploads limits the number of files allowed in a single POST (not the size). It defaults to 20 I think. See https://www.php.net/manual/en/ini.core.php#ini.max-file-uploads.

[BirenRathod]

@johnvanbreda Yes, those are the numbers. Sorry I have put MB with it, so it is 10 and 20.

[BirenRathod]

@johnvanbreda Sorry it is 50 on iRecord and 20 on warehouse.

[johnvanbreda]

@BirenRathod This seems to be the only setting that might make a difference - can we test it somehow?

[kazlauskis]

Hi both, I have tested the submission and I could now successfully upload a record with 25 images. Did you change anything on the server?

@DavidRoy you can try uploading your survey now.

[BirenRathod]

@kazlauskis It actually wrong value with M stored in the PHP config file, so it considered the default value instead.

[DavidRoy]

Yes, my submission now went through @BirenRathod - so I understand the cause of this problem. Are you saying that there was an incorrect value stored in the PHP config file? Can you now confirm the size limits please.

[BirenRathod]

@DavidRoy

It was before 50m was not working, so I removed M , so it is 50 for all websites on BRC server and 20 on Indicia warehouse.

[DavidRoy]

@BirenRathod so the limit for app data submissions is 20Mb? @kazlauskis is there anyway to trap for this limit being reached so we can warn users?

[kazlauskis]

Thanks all, I am closing this then.

[kazlauskis]

Yes, we could roughly calculate the size and so prevent exceeding the limit on the app side. We could also resize the attached images within the app, so we don't try to upload 5Mb photos. A better solution here would be to be able to upload the media individually. This would allow to upload videos and other larger attachments then.

[DavidRoy]

Thanks. Do you want to raise new issues for these suggestions, so we don't forget them.

[BirenRathod]

@DavidRoy limit on iRecord website is 100MB & on indicia warehouse is 15MB

[DavidRoy]

Still getting reports of issues - https://www.brc.ac.uk/irecord/node/8053#comment-5601

[kazlauskis]

I'm not sure if this is still an issue so closing it for now.