search

farmOS / field-kit

A modular, offline-first companion app to farmOS.
https://farmOS.org
GNU General Public License v3.0
60 stars 39 forks source link

Field Kit Logs With Images Not Uploading to Server #365

Closed vital-agro closed 4 years ago

vital-agro commented 4 years ago

After syncing three test logs, I noticed that only the logs that were created offline were uploaded to our farmOS server. See screenshots attached.

Logs were created using iOS PWA saved to the home screen.FK version 0.5.1 on iOS 13.6 with safari 13.

vital-agro commented 4 years ago

IMG_1659 IMG_1660

jgaehring commented 4 years ago

Thanks @vital-agro. Am I correct that you're also getting duplicate logs sent to your server? Are you getting any error messages in Field Kit when you try to sync?

vital-agro commented 4 years ago

Yes, I synced all the logs twice and the test cache log was duplicated. Also the “test withour service” didnt upload. I was next to an Arable Mark weather station during data collection and upload, could that be interfering? There were no error messages.

Mason

On Thu, Jul 30, 2020 at 10:26 AM Jamie Gaehring notifications@github.com wrote:

Thanks @vital-agro https://github.com/vital-agro. Am I correct that you're also getting duplicate logs sent to your server? Are you getting any error messages in Field Kit when you try to sync?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/farmOS/farmOS-client/issues/365#issuecomment-666543738, or unsubscribe https://github.com/notifications/unsubscribe-auth/APXNOFXR5LYSGQ544XTIK3DR6GUKRANCNFSM4POEM2IA .

-- Mason Lanphear Founder, Vital Agronomics LLC Email: mason@vital-agronomics.com Cell: (206)-422-2911

jgaehring commented 4 years ago

Also, I just noticed some bizarre behavior when testing on iPad right now, after coming out of airplane mode, the sync button was non-responsive for a moment, no error messages. This was resolved when I went to the home screen and then reopened the app. Perhaps this might work for you?

I synced all the logs twice and the test cache log was duplicated.

Hm, this makes me think there could be an issue with a database migration, possibly some corrupt data. Were you on a different version before 0.5.1? If so, do you happen to recall the version number?

If you don't have any data you need to save, it might help to logout and login again. That will clear the database and start fresh.

vital-agro commented 4 years ago

I think I was using 0.5.1. I logged in and out but logs still aren't uploading, so it may be a data migration issue.

Thanks

vital-agro commented 4 years ago

Jamie: I reinstalled the home screen PWA and now the logs are showing up on the server, but after turning cellular service and WiFi off and on, any new logs won't upload.

jgaehring commented 4 years ago

Hm, I'm having trouble reproducing. The only issue I'm experiencing by coming in and out of airplane mode is the behavior I explained above, where the sync button, and all other buttons, are non-responsive for a time, until I go back to the home screen and reopen the app. Could this be what happens to you?

When you say new logs won't upload, what is the exact behavior? Do the cloud icons for each unsynced log become spinners? For how long do they spin? Or are you seeing no visual response to tapping the sync button? And you're still seeing no error messages?

vital-agro commented 4 years ago

I deleted the PWA from the home screen and then reinstalled it, which enabled me to upload several logs but once I went offline and then online (turning off WiFi and Cellular) I could no longer sync logs. I don't get unresponsive sync icons, and it runs the sync icon for about a second. Still no error messages

I logged out and reinstalled the home screen app again to reproduce the issue and it's definitely the act of turning off cell service or WiFi that triggers the issue.

vital-agro commented 4 years ago

Update: after several reinstall attempts it seems to be working online and offline for now.

vital-agro commented 4 years ago

Jamie: I think I found the issue. It only fails to upload when I try to upload an image that was captured via the app while creating the log. I was able to upload previously captured images.

Let me know if you can reproduce that.

jgaehring commented 4 years ago

Oh that's interesting. Hm, I still can't reproduce. Are there any other specific attributes on the log? What log type are you sending?

vital-agro commented 4 years ago

It's an activity log with gps coordinates and an image attachment.

vital-agro commented 4 years ago

Hmm. Now I can't upload logs with any photos.

jgaehring commented 4 years ago

Hmm, very strange. Do you have many logs with photos in the app? At this point we're not doing much to optimize how they're stored, so there could be a limit to how many you can have stored at one time. I don't know how that would affect uploading, but maybe if you tried deleting some first? Sorry, kinda grasping at straws here, since I still haven't reproduced what you're seeing. :confused:

vital-agro commented 4 years ago

I deleted all cached logs on my device and I still can't upload any logs with images, so I don't think that's the issue. I was able to upload more logs without images but they start to duplicate once I try to add logs with images. Also the app tends to be unresponsive at times until I close and reopen it.

jgaehring commented 4 years ago

I'm starting to wonder if this might be something to do with how your server is configured. @mstenta, do you think there's any reason why restws_file might be failing to accept images?

Also the app tends to be unresponsive at times until I close and reopen it.

Ok, yea, this is what I see from time to time too. Doing a little research now, it seems like this is a known bug: https://bugs.webkit.org/show_bug.cgi?id=211018. Bummer :confused:

Btw, thanks for sticking with this, Mason, I really appreciate the feedback. Hopefully we can get this resolved soon!

vital-agro commented 4 years ago

@paul121 mentioned something about our server configuration in relation to this issue

mstenta commented 4 years ago

Ah hmm, nothing comes to mind - but I wonder... @vital-agro it might be worth testing uploading to test.farmos.net, and see if that behaves the same or not. @paul121 can give you credentials to it. If you have the chance, that might be a good way to see if it's a server issue or a Field Kit issue.

vital-agro commented 4 years ago

Thanks @mstenta, looks like it's a server issue. I was able to upload image logs to the test server.

mstenta commented 4 years ago

Good to hear that it was not a Field Kit issue! Happy to help figure out the server issue!

paul121 commented 4 years ago

I'm able to replicate this using latest FK in Firefox on Android. It's worth noting that the farmOS server is still on v1.4. This should be OK with FK v0.5.1 though?

I just applied this commit to allow OPTIONS to `farm/areas/geojson' thinking it might be related (if that request fails, would the rest of the sync process fail?) but it doesn't seem to have fixed the problem.

What's interesting are the farmOS server logs... when I create a log in FK that has an image, no POST request is ever sent to the server. Logs without images go through fine. See logs below. This seems weird - @mstenta anything server side that might be causing this?

172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "OPTIONS /taxonomy_term.json?bundle=farm_quantity_units&page=0 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "OPTIONS /taxonomy_vocabulary.json HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "OPTIONS /taxonomy_term.json?bundle=farm_log_categories&page=0 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "OPTIONS /log.json?done=0&log_owner=3&page=0 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "OPTIONS /farm_asset.json?&archived=0&page=0 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "GET /log.json?done=0&log_owner=3&page=0 HTTP/1.0" 200 1023 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "GET /taxonomy_vocabulary.json HTTP/1.0" 200 2270 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "GET /taxonomy_term.json?bundle=farm_quantity_units&page=0 HTTP/1.0" 200 2616 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "GET /taxonomy_term.json?bundle=farm_log_categories&page=0 HTTP/1.0" 200 4322 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:39 +0000] "OPTIONS /log HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:39 +0000] "OPTIONS /taxonomy_term.json?vocabulary=3&&page=0 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:38 +0000] "GET /farm_asset.json?&archived=0&page=0 HTTP/1.0" 200 113806 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:40 +0000] "OPTIONS /farm_asset.json?&archived=0&page=1 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:39 +0000] "GET /taxonomy_term.json?vocabulary=3&&page=0 HTTP/1.0" 200 132359 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:40 +0000] "OPTIONS /taxonomy_term.json?vocabulary=3&&page=1 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:40 +0000] "GET /farm_asset.json?&archived=0&page=1 HTTP/1.0" 200 109243 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:42 +0000] "OPTIONS /farm_asset.json?&archived=0&page=2 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:40 +0000] "GET /taxonomy_term.json?vocabulary=3&&page=1 HTTP/1.0" 200 142382 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:42 +0000] "OPTIONS /taxonomy_term.json?vocabulary=3&&page=2 HTTP/1.0" 200 582 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:42 +0000] "GET /farm_asset.json?&archived=0&page=2 HTTP/1.0" 200 18210 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
172.18.0.3 - - [31/Jul/2020:15:31:42 +0000] "GET /taxonomy_term.json?vocabulary=3&&page=2 HTTP/1.0" 200 51962 "https://farmos.app/" "Mozilla/5.0 (Android 10; Mobile; rv:68.0) Gecko/68.0 Firefox/68.0"
paul121 commented 4 years ago

Aha! @mstenta diagnosed the problem and prescribed a solution :clap:

The problem was file uploads were limited to 1MB and being blocked by the proxy. The images we were testing were > 5MB, but I was able to upload a 500kb image from Field Kit.

To fix I made the following changes on the server:

# Allow large file uploads.
client_max_body_size 50M;

https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size

Add the following PHP settings:

 'upload_max_filesize=50M';
 'post_max_size=50M';