search

inaturalist / iNaturalistAndroid

Android app for iNaturalist.org
https://market.android.com/details?id=org.inaturalist.android
MIT License
159 stars 57 forks source link

app crashing when obs json is too large #312

Closed kueda closed 4 years ago

kueda commented 7 years ago

I think an observation's JSON is too large the app will crash when loading comments and IDs. This was happening when you try to view https://www.inaturalist.org/observations/5407184.json in the app (it's not now b/c I removed a lot of the photos). You could definitely argue that this response is absurd with a ton of duplicate taxon and photos records, and you would be right, but regardless, the app should not choke on JSON just b/c it's too big unless it's absurdly big, like 5+ MB or something. Or maybe I'm mis-diagnosing this, but I'm definitely getting a crash on my 5X, build 199. I do not get that crash with build 203 after having removed the photos. Yes I should have tested the most recent version before removing the photos. I am a bad bug filer.

The user who reported this is the one who owns this observation, and they also say that when they try to view it in the app, the app hangs and posts an older version of their ID (you can see the sequence of replaced IDs on their observation). I'm not entirely sure that's related, but maybe.

Anyway, if there is some kind of cap on the size of JSON we can load, can we remove it?

Log data:

03-24 21:56:48.289 16607 19244 D INaturalistService: RESP: [{"id":5406499,"observed_on":"2015-09-17","description":null,"latitude":"30.1886227692","longitude":"-97.7197442204","map_scale":null,"timeframe":null,"species_guess":"Rabid Wolf Spider","user_id":370785,"taxon_id":147930,"created_at":"2017-03-19T22:00:43.804-07:00","updated_at":"2017-03-20T18:17:59.438-07:00","place_guess":"Austin, TX","id_please":false,"observed_on_string":"2015-09-17 12:56:00 PM CDT","iconic_taxon_id":47119,"num_identification_agreements":1,"num_identification_disagreements":0,"time_observed_at":"2015-09-17T10:56:00.000-07:00","time_zone":"Central Time (US \u0026 Canada)","location_is_exact":false,"delta":false,"positional_accuracy":86,"geoprivacy":null,"quality_grade":"research","positioning_method":"gps","positioning_device":"gps","out_of_range":null,"license":null,"uri":"https://www.inaturalist.org/observations/5406499","observation_photos_count":1,"comments_count":0,"zic_time_zone":"America/Chicago","oauth_application_id":2,"observation_sounds_count":0,"identifications_count":6,"captive":false,"community_taxon_id":147930,"site_id":1,"old_uuid":null,"public_positional_accuracy":86,"mappable":true,"cached_votes_total":0,"last_indexed_at":"2017-03-22T17:56:03.899-07:00","uuid":"555394fc-587a-4d34-b784-52ab3401878c","short_description":null,"user_login":"star3","iconic_taxon_name":"Arachnida","tag_list":[],"faves_count":0,"created_at_utc":"2017-03-20T05:00:43.804Z","updated_at_utc":"2017-03-21T01:17:59.438Z","time_observed_at_utc":"2015-09-17T17:56:00.000Z","coordinates_obscured":false,"taxon":{"id":147930,"name":"Rabidosa rabida","rank":"species","ancestry":"48460/1/47120/245097/47119/47118/120474/342614/367161/47416/147238","common_name":{"id":248732,"name":"Rabid Wolf Spider","is_valid":true,"lexicon":"English"}},"iconic_taxon":{"id":47119,"name":"Arachnida","rank":"class","rank_level":50,"ancestry":"48460/1/47120/245097"},"user":{"login":"star3","user_icon_url":null},"photos":[{"id":6644200,"user_id":370785,"native_photo_id":"6644200","square_url":"https://static.inaturalist.org/photos/6644200/square.jpeg?1489986051","thumb_url":"https://static.inaturalist.org/photos/6644200/thumb.jpeg?1489986051","small_url":"https://static.inaturalist.org/photos/6644200/small.jpeg?1489986051","medium_url":"https://static.inaturalist.org/photos/6644200/medium.jpeg?1489986051","large_url":"https://static.inaturalist.org/photos/6644200/large.jpeg?1489986051","created_at":"2017-03-19T22:00:49.668-07:00","updated_at":"2017-03-19T22:00:53.251-07:00","native_page_url":"https://www.inaturalist.org/photos/6644200","native_username":"star3","native_realname":"Star Donovan","license":0,"subtype":null,"native_original_image_url":null,"license_code":"C","attribution":"(c) Star Donovan, all rights reserved","license_name":"Copyright","license_url":"http://en.wikipedia.org/wiki/Copyright","type":"LocalPhoto"}],"observation_photos":[{"id":6301732,"observation_id":5406499,"photo_id":6644200,"position":0,"created_at":"2017-03-19T22:00:50.339-07:00","updated_at":"2017-03-19T22:00:50.339-07:00","old_uuid":null,"uuid":"86b29f32-5767-4004-919e-4886c5bf84ab","photo":{"id":6644200,"user_id":370785,"native_photo_id":"6644200","square_url":"https://static.inaturalist.org/photos/6644200/square.jpeg?1489986051","thumb_url":"https://static.inaturalist.org/photos/6644200/thumb.jpeg?1489986051","small_url":"https://static.inaturalist.org/photos/6644200/small.jpeg?1489986051","medium_url":"https://static.inaturalist.org/photos/6644200/medium.jpeg?1489986051","large_url":"https://static.inaturalist.org/photos/6644200/large.jpeg?1489986051","created_at":"2017-03-19T22:00:49.668-07:00","updated_at":"2017-03-19T22:00:53.251-07:00","native_page_url":"https://www.inaturalist.org/photos/6644200","native_username":"star3","native_realname":"Star Donovan","license":0,"subtype":null,"native_original_image_url":null,"license_name":"Copyright","license_url":"http://en.wikipedia.org/wiki/Copyright","attribution":"(c) Star Donovan, all rights reserved","type":"LocalPhoto"}}]},{"id":5406410,"observed_on"
03-24 21:56:48.334 16607 19244 D INaturalistService: Service: get_observation
03-24 21:56:48.336 16607 19244 D INaturalistService: URL: get - https://www.inaturalist.org/observations/5407184.json?locale=en (null)
03-24 21:56:48.488 16607 16607 W FlurryAgent: End session with context: org.inaturalist.android.INaturalistMapActivity@14bbb16 count:1
03-24 21:56:48.500 16607 16607 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 987252)
03-24 21:56:48.503 16607 16607 E ActivityThread: App sent too much data in instance state, so it was ignored
03-24 21:56:48.503 16607 16607 E ActivityThread: android.os.TransactionTooLargeException: data parcel size 987252 bytes
03-24 21:56:48.503 16607 16607 E ActivityThread:    at android.os.BinderProxy.transactNative(Native Method)
03-24 21:56:48.503 16607 16607 E ActivityThread:    at android.os.BinderProxy.transact(Binder.java:615)
03-24 21:56:48.503 16607 16607 E ActivityThread:    at android.app.ActivityManagerProxy.activityStopped(ActivityManagerNative.java:3636)
03-24 21:56:48.503 16607 16607 E ActivityThread:    at android.app.ActivityThread$StopInfo.run(ActivityThread.java:3773)
03-24 21:56:48.503 16607 16607 E ActivityThread:    at android.os.Handler.handleCallback(Handler.java:751)
03-24 21:56:48.503 16607 16607 E ActivityThread:    at android.os.Handler.dispatchMessage(Handler.java:95)
03-24 21:56:48.503 16607 16607 E ActivityThread:    at android.os.Looper.loop(Looper.java:154)
03-24 21:56:48.503 16607 16607 E ActivityThread:    at android.app.ActivityThread.main(ActivityThread.java:6119)
03-24 21:56:48.503 16607 16607 E ActivityThread:    at java.lang.reflect.Method.invoke(Native Method)
03-24 21:56:48.503 16607 16607 E ActivityThread:    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:886)
03-24 21:56:48.503 16607 16607 E ActivityThread:    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:776)
03-24 21:56:48.731 16607 16612 I art     : Do partial code cache collection, code=221KB, data=230KB
03-24 21:56:48.732 16607 16612 I art     : After code cache collection, code=198KB, data=208KB
03-24 21:56:48.732 16607 16612 I art     : Increasing code cache capacity to 1024KB
03-24 21:56:52.053 16607 19244 D INaturalistService: RESP: {"id":5407184,"observed_on":"2016-04-07","description":null,"latitude":"30.1950011529","longitude":"-97.7162627131","map_scale":null,"timeframe":null,"species_guess":"Flaxes","user_id":370785,"taxon_id":51309,"created_at":"2017-03-20T07:37:34.493+01:00","updated_at":"2017-03-23T06:07:56.365+01:00","place_guess":"4200 Smith School Drive Austin, TX 78744","id_please":false,"observed_on_string":"2016-04-07 12:09:00 PM CDT","iconic_taxon_id":47126,"num_identification_agreements":1,"num_identification_disagreements":0,"time_observed_at":"2016-04-07T18:09:00.000+01:00","time_zone":"Central Time (US \u0026 Canada)","location_is_exact":false,"delta":false,"positional_accuracy":344,"geoprivacy":null,"quality_grade":"needs_id","positioning_method":"gps","positioning_device":"gps","out_of_range":null,"license":null,"uri":"https://www.inaturalist.org/observations/5407184","observation_photos_count":2,"comments_count":1,"zic_time_zone":"America/Chicago","oauth_application_id":2,"observation_sounds_count":0,"identifications_count":9,"captive":false,"community_taxon_id":51309,"site_id":1,"old_uuid":null,"public_positional_accuracy":344,"mappable":true,"cached_votes_total":0,"last_indexed_at":"2017-03-23T06:24:39.561+01:00","uuid":"255d4efe-3968-4c59-be04-e363d6f61a3e","user_login":"star3","iconic_taxon_name":"Plantae","captive_flag":false,"created_at_utc":"2017-03-20T06:37:34.493Z","updated_at_utc":"2017-03-23T05:07:56.365Z","time_observed_at_utc":"2016-04-07T17:09:00.000Z","faves_count":0,"coordinates_obscured":false,"user":{"id":370785,"login":"star3","name":"Star Donovan","created_at":"2016-11-17T20:03:36.419+01:00","updated_at":"2017-03-22T23:57:24.900+01:00","time_zone":null,"description":null,"life_list_id":516782,"observations_count":118,"identifications_count":18,"journal_posts_count":0,"life_list_taxa_count":137,"icon_url":"http://graph.facebook.com/10211134117611402/picture?type=square","uri":"http://www.inaturalist.org/users/370785","locale":"en","site_id":null,"place_id":null,"spammer":false,"spam_count":0,"last_active":"2017-03-24","subscriptions_suspended_at":null,"test_groups":null,"latitude":30.0787,"longitude":-97.8341,"lat_lon_acc_admin_level":2,"icon_file_name":null,"icon_content_type":null,"icon_file_size":null,"icon_updated_at":null,"user_icon_url":null,"medium_user_icon_url":null,"original_user_icon_url":null},"observation_field_values":[{"id":4030105,"observation_id":5407184,"observation_field_id":379,"value":"unknown","created_at":"2017-03-20T07:38:15.029+01:00","updated_at":"2017-03-20T07:38:15.029+01:00","user_id":370785,"updater_id":370785,"uuid":"c24846be-0bfd-41e0-991a-989c81fce2f0","observation_field":{"name":"Invasive?"}}],"project_observations":[{"id":7982657,"project_id":544,"observation_id":5407184,"created_at":"2017-03-20T07:38:16.138+01:00","updated_at":"2017-03-20T07:38:16.138+01:00","curator_identification_id":null,"tracking_code":null,"user_id":370785,"uuid":"4255f83f-69b2-4202-8347-9ee0f466274c","project":{"id":544,"title":"Plants of Texas","description":"The purpose of this project is to broaden the understanding of Texas plant communities - both native and non-native - and contribute to knowledge of their diversity and range. It is hoped that the observations within this project will lead to a greater appreciation of the botanical Texas, while also helping guide resources toward conservation efforts.   \r\n\r\nWhile this project was started by a citizen scientist and Texas Master Naturalist as a hobby project, we encourage the use of the project and data by research and conservation professionals working in all areas of Texas botany. Also, observations of Species of Greatest Conservation Need (SGCN) may be invited to other appropriate projects, such as the TPWD-curated project, Texas Botanical Treasures at http://www.inaturalist.org/projects/texas-botanical-treasures  so that conservation planning and resources can be directed at the plant community as needed. \r\n\r\nAs such, we need accurate location data, good photos, and - hopefully - s
03-24 21:56:52.122 16607 19244 E INaturalistService: Failed to create JSONArray, JSONException: org.json.JSONException: Value {"id":5407184,"observed_on":"2016-04-07","description":null,"latitude":"30.1950011529","longitude":"-97.7162627131","map_scale":null,"timeframe":null,"species_guess":"Flaxes","user_id":370785,"taxon_id":51309,"created_at":"2017-03-20T07:37:34.493+01:00","updated_at":"2017-03-23T06:07:56.365+01:00","place_guess":"4200 Smith School Drive Austin, TX 78744","id_please":false,"observed_on_string":"2016-04-07 12:09:00 PM CDT","iconic_taxon_id":47126,"num_identification_agreements":1,"num_identification_disagreements":0,"time_observed_at":"2016-04-07T18:09:00.000+01:00","time_zone":"Central Time (US & Canada)","location_is_exact":false,"delta":false,"positional_accuracy":344,"geoprivacy":null,"quality_grade":"needs_id","positioning_method":"gps","positioning_device":"gps","out_of_range":null,"license":null,"uri":"https:\/\/www.inaturalist.org\/observations\/5407184","observation_photos_count":2,"comments_count":1,"zic_time_zone":"America\/Chicago","oauth_application_id":2,"observation_sounds_count":0,"identifications_count":9,"captive":false,"community_taxon_id":51309,"site_id":1,"old_uuid":null,"public_positional_accuracy":344,"mappable":true,"cached_votes_total":0,"last_indexed_at":"2017-03-23T06:24:39.561+01:00","uuid":"255d4efe-3968-4c59-be04-e363d6f61a3e","user_login":"star3","iconic_taxon_name":"Plantae","captive_flag":false,"created_at_utc":"2017-03-20T06:37:34.493Z","updated_at_utc":"2017-03-23T05:07:56.365Z","time_observed_at_utc":"2016-04-07T17:09:00.000Z","faves_count":0,"coordinates_obscured":false,"user":{"id":370785,"login":"star3","name":"Star Donovan","created_at":"2016-11-17T20:03:36.419+01:00","updated_at":"2017-03-22T23:57:24.900+01:00","time_zone":null,"description":null,"life_list_id":516782,"observations_count":118,"identifications_count":18,"journal_posts_count":0,"life_list_taxa_count":137,"icon_url":"http:\/\/graph.facebook.com\/10211134117611402\/picture?type=square","uri":"http:\/\/www.inaturalist.org\/users\/370785","locale":"en","site_id":null,"place_id":null,"spammer":false,"spam_count":0,"last_active":"2017-03-24","subscriptions_suspended_at":null,"test_groups":null,"latitude":30.0787,"longitude":-97.8341,"lat_lon_acc_admin_level":2,"icon_file_name":null,"icon_content_type":null,"icon_file_size":null,"icon_updated_at":null,"user_icon_url":null,"medium_user_icon_url":null,"original_user_icon_url":null},"observation_field_values":[{"id":4030105,"observation_id":5407184,"observation_field_id":379,"value":"unknown","created_at":"2017-03-20T07:38:15.029+01:00","updated_at":"2017-03-20T07:38:15.029+01:00","user_id":370785,"updater_id":370785,"uuid":"c24846be-0bfd-41e0-991a-989c81fce2f0","observation_field":{"name":"Invasive?"}}],"project_observations":[{"id":7982657,"project_id":544,"observation_id":5407184,"created_at":"2017-03-20T07:38:16.138+01:00","updated_at":"2017-03-20T07:38:16.138+01:00","curator_identification_id":null,"tracking_code":null,"user_id":370785,"uuid":"4255f83f-69b2-4202-8347-9ee0f466274c","project":{"id":544,"title":"Plants of Texas","description":"The purpose of this project is to broaden the understanding of Texas plant communities - both native and non-native - and contribute to knowledge of their diversity and range. It is hoped that the observations within this project will lead to a greater appreciation of the botanical Texas, while also helping guide resources toward conservation efforts.   \r\n\r\nWhile this project was started by a citizen scientist and Texas Master Naturalist as a hobby project, we encourage the use of the project and data by research and conservation professionals working in all areas of Texas botany. Also, observations of Species of Greatest Conservation Need (SGCN) may be invited to other appropriate projects, such as the TPWD-curated project, Texas Botanical Treasures at http:\/\/www.inaturalist.org\/projects\/texas-botanical-treasures  so that conservation planning and resources can be directed at the plant community as needed. \r
03-24 21:56:52.257 16607 19244 D INaturalistService: Service: get_observation
03-24 21:56:52.258 16607 16607 D AndroidRuntime: Shutting down VM
03-24 21:56:52.260 16607 19244 D INaturalistService: URL: get - https://www.inaturalist.org/observations/5407184.json?locale=en (null)
03-24 21:56:52.261 16607 16607 W System.err: android.app.RemoteServiceException: can't deliver broadcast
03-24 21:56:52.261 16607 16607 W System.err:    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1650)
03-24 21:56:52.261 16607 16607 W System.err:    at android.os.Handler.dispatchMessage(Handler.java:102)
03-24 21:56:52.261 16607 16607 W System.err:    at android.os.Looper.loop(Looper.java:154)
03-24 21:56:52.261 16607 16607 W System.err:    at android.app.ActivityThread.main(ActivityThread.java:6119)
03-24 21:56:52.261 16607 16607 W System.err:    at java.lang.reflect.Method.invoke(Native Method)
03-24 21:56:52.261 16607 16607 W System.err:    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:886)
03-24 21:56:52.261 16607 16607 W System.err:    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:776)
03-24 21:56:52.271 16607 16607 W FlurryAgent: Error logged: uncaught
03-24 21:56:52.275 16607 16607 W FlurryAgent: End session with context: org.inaturalist.android.ObservationViewerActivity@41602aa count:0
03-24 21:56:52.278 16607 16607 W FlurryAgent: Finalize session
03-24 21:56:52.499 16607 16607 E AndroidRuntime: FATAL EXCEPTION: main
03-24 21:56:52.499 16607 16607 E AndroidRuntime: Process: org.inaturalist.android, PID: 16607
03-24 21:56:52.499 16607 16607 E AndroidRuntime: android.app.RemoteServiceException: can't deliver broadcast
03-24 21:56:52.499 16607 16607 E AndroidRuntime:    at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1650)
03-24 21:56:52.499 16607 16607 E AndroidRuntime:    at android.os.Handler.dispatchMessage(Handler.java:102)
03-24 21:56:52.499 16607 16607 E AndroidRuntime:    at android.os.Looper.loop(Looper.java:154)
03-24 21:56:52.499 16607 16607 E AndroidRuntime:    at android.app.ActivityThread.main(ActivityThread.java:6119)
03-24 21:56:52.499 16607 16607 E AndroidRuntime:    at java.lang.reflect.Method.invoke(Native Method)
03-24 21:56:52.499 16607 16607 E AndroidRuntime:    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:886)
03-24 21:56:52.499 16607 16607 E AndroidRuntime:    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:776)
03-24 21:56:53.069 16607 19292 W FlurryAgent: FlurryDataSender: report 209d7eda-7123-4a59-9eab-e9e5ff31b35c sent. HTTP response: 200
03-24 21:56:53.619 16607 16629 I Fabric  : Crashlytics report upload complete: 58D5F0E80032-0001-40DF-46BDB01DD930.cls
budowski commented 7 years ago

We can transmit this data in other ways (it's another method of transferring results from the INatService class - we've implemented before for other APIs). Will fix

kueda commented 5 years ago

FWIW, this seems like it's still happening, only on Google devices and only in Android 9: https://fabric.io/inaturalist/android/apps/org.inaturalist.android/issues/5c1430b6f8b88c2963db7913?time=last-thirty-days. Not sure if it's really impacting users.

kueda commented 5 years ago

Definitely not fixed as it's still happening in 335 and 337. The question is, can it be fixed or is this just due to giant JSON responses from the API?

kueda commented 4 years ago

FWIW, this is still happening in build 370, but it remains unclear whether this is fixable. Smaller API responses are something we're working on in the next version of the API.