search

PnX-SI / gn_mobile_occtax

Application mobile pour la saisie dans le module Occtax de GeoNature
GNU General Public License v3.0
13 stars 5 forks source link

La synchro plante à cause d'un timeout sur l'appel d'API `/api/synthese/color_taxon` #252

Closed lpofredc closed 1 month ago

lpofredc commented 6 months ago

Version de l'application

Version d'Occtax-mobile affectée par le bug : 2.6.2.32200 Version de GeoNature utilisée : 2.13.0

Terminal et Version Android

Marque et modèle du terminal : OnePlus 5 Version d'Android : 13 (LineageOS 20-20231205-migrog-dumpling)

Description du bug et comportement attendu

La synchro de l'appli s'interrompt avec une erreur lié à un timeout sur l'API color_taxon. Sans doute une conséquence des pb de perfs du dispositif des couleurs taxon dans GeoNature (vue vs table ou vue matérialisée, cf. https://github.com/PnX-SI/gn_mobile_occtax/issues/232, https://github.com/PnX-SI/GeoNature/issues/2699).

Instance de 33 millions de données avec 360 millions de lignes dans gn_synthese.cor_area_synthese.

J'ai tenté la désactivation de la synchro des couleurs de taxon comme évoqué dans ce patch https://github.com/PnX-SI/gn_mobile_core/commit/bd9685c7f5ee1c01972275c803ee173f34b0bf15 mais il semble que ce ne soit pas encore répercuté dans l'appli.

J'ai donc provisoirement désactivé la fonctionnalité en modifiant la vue comme suit (WHERE FALSE pour vider son contenu) 

CREATE VIEW v_area_taxon(cd_nom, id_area, nb_obs, last_date) AS
SELECT s.cd_nom,
       c.id_area,
       count(DISTINCT s.id_synthese) AS nb_obs,
       max(s.date_min)               AS last_date
FROM gn_synthese.synthese s
         JOIN gn_synthese.cor_area_synthese c ON s.id_synthese = c.id_synthese
         JOIN ref_geo.l_areas la ON la.id_area = c.id_area AND la.id_type =
                                                               ref_geo.get_id_area_type(gn_commons.get_default_parameter('occtaxmobile_area_type'::TEXT)::CHARACTER VARYING)
WHERE FALSE
GROUP BY c.id_area, s.cd_nom;

Comment reproduire

settings.json

{
  "area_observation_duration": 365,
  "sync": {
    "geonature_url": "https://geonature.lpo-aura.org",
    "taxhub_url": "https://geonature.lpo-aura.org/taxhub",
    "gn_application_id": 3,
    "observers_list_id": 1,
    "taxa_list_id": 100,
    "code_area_type": "M10",
    "page_size": 10000
  },
  "map": {
    "show_scale": true,
    "show_compass": true,
    "max_bounds": [
      [52.0, -6.0],
      [41.0, 9.0]
    ],
    "center": [46.0, 3.0],
    "start_zoom": 10.0,
    "min_zoom": 8.0,
    "max_zoom": 19.0,
    "min_zoom_editing": 12.0,
    "base_path": "Offline_maps",
    "layers": [
      {
        "label": "IGN: plan v2",
        "source": "https://wxs.ign.fr/essentiels/geoportail/wmts?REQUEST=GetTile&SERVICE=WMTS&VERSION=1.0.0&STYLE=normal&TILEMATRIXSET=PM&FORMAT=image/png&LAYER=GEOGRAPHICALGRIDSYSTEMS.PLANIGNV2"
      },
      {
        "label": "IGN: ortho",
        "source": "https://wxs.ign.fr/ortho/geoportail/wmts?REQUEST=GetTile&SERVICE=WMTS&VERSION=1.0.0&STYLE=normal&TILEMATRIXSET=PM&FORMAT=image/jpeg&LAYER=ORTHOIMAGERY.ORTHOPHOTOS"
      },
      {
        "label": "OpenStreetMap",
        "source": [
          "https://a.tile.openstreetmap.org",
          "https://b.tile.openstreetmap.org",
          "https://c.tile.openstreetmap.org"
        ]
      },
      {
        "label": "OpenTopoMap",
        "source": [
          "https://a.tile.opentopomap.org",
          "https://b.tile.opentopomap.org",
          "https://c.tile.opentopomap.org"
        ]
      },
      {
        "label": "IGN plan",
        "source": "plan.mbtiles"
      },
      {
        "label": "IGN ortho",
        "source": "ortho.mbtiles"
      },
      {
        "label": "Mailles 5x5",
        "source": "mailles.geojson"
      }
    ]
  },
  "input": {
    "date": {
      "enable_end_date": true,
      "enable_hours": true
    }
  }
}

Logs

12:27:47.701 INFO: [fr.geonature.occtax.MainApplication] starting fr.geonature.occtax2...
12:27:47.702 INFO: [fr.geonature.occtax.MainApplication] logs directory: '/storage/emulated/0/Android/data/fr.geonature.occtax2/logs'
12:27:47.702 INFO: [fr.geonature.occtax.MainApplication] internal storage: 'MountPoint(mountPath=/storage/emulated/0, storageType=INTERNAL)'
12:27:47.702 INFO: [fr.geonature.occtax.MainApplication] external storage: 'null'
12:27:47.763 INFO: [fr.geonature.datasync.features.settings.usecase.UpdateSettingsFromRemoteUseCase] loading app configuration...
12:27:47.763 INFO: [fr.geonature.datasync.settings.DataSyncSettingsFileDataSourceImpl] loading data sync settings from '/storage/emulated/0/Android/data/fr.geonature.occtax2/settings_occtax.json'...
12:27:47.769 INFO: [fr.geonature.commons.data.DatabaseModule] loading local database '/storage/emulated/0/Android/data/fr.geonature.occtax2/databases/data.db'...
12:27:47.786 INFO: [fr.geonature.datasync.api.GeoNatureAPIClientImpl] set server base URLs (GeoNature: 'https://geonature.lpo-aura.org', TaxHub: 'https://geonature.lpo-aura.org/taxhub')...
12:27:47.791 INFO: [fr.geonature.datasync.features.settings.usecase.UpdateSettingsFromRemoteUseCase] updating app configuration from 'https://geonature.lpo-aura.org'...
12:27:47.860 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/api/gn_commons/t_mobile_apps
12:27:47.863 INFO: [fr.geonature.datasync.packageinfo.worker.CheckInputsToSynchronizeWorker] available inputs to synchronize: 0
12:27:48.144 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/api/gn_commons/t_mobile_apps (282ms, unknown-length body)
12:27:48.150 INFO: [fr.geonature.datasync.packageinfo.PackageInfoRepositoryImpl] updating settings for 'fr.geonature.occtax2'...
12:27:48.163 INFO: [fr.geonature.datasync.packageinfo.io.AppSettingsJsonWriter] updating app settings '/storage/emulated/0/Android/data/fr.geonature.occtax2/settings_occtax.json'
12:27:48.164 INFO: [fr.geonature.datasync.settings.DataSyncSettingsFileDataSourceImpl] loading data sync settings from '/storage/emulated/0/Android/data/fr.geonature.occtax2/settings_occtax.json'...
12:27:48.165 INFO: [fr.geonature.datasync.api.GeoNatureAPIClientImpl] set server base URLs (GeoNature: 'https://geonature.lpo-aura.org', TaxHub: 'https://geonature.lpo-aura.org/taxhub')...
12:27:48.166 INFO: [fr.geonature.datasync.features.settings.usecase.UpdateSettingsFromRemoteUseCase] app configuration successfully updated from 'https://geonature.lpo-aura.org'
12:27:48.167 INFO: [fr.geonature.occtax.ui.home.HomeActivity] reloading settings after update...
12:27:48.168 INFO: [fr.geonature.commons.settings.AppSettingsManagerImpl] loading settings from URI 'content://fr.geonature.occtax2.provider/settings/settings_occtax.json'...
12:27:48.171 INFO: [fr.geonature.occtax.settings.io.OnAppSettingsJsonReaderListenerImpl] input date settings loaded ('start': datetime, 'end': datetime)
12:27:48.173 INFO: [fr.geonature.occtax.ui.home.HomeActivity] app settings successfully loaded
12:27:48.174 INFO: [fr.geonature.datasync.sync.DataSyncViewModel] starting local data synchronization...
12:27:48.212 INFO: [fr.geonature.datasync.sync.DataSyncViewModel] configure data sync periodic worker (repeat interval: 7d, with additional data: true)...
12:27:48.251 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] starting local data synchronization from 'https://geonature.lpo-aura.org' (with additional data: true)...
12:27:48.252 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] synchronize dataset...
12:27:48.253 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/api/meta/datasets?fields=modules
12:27:56.412 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/api/meta/datasets?fields=modules (8158ms, unknown-length body)
12:27:57.142 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] dataset to update: 74
12:27:57.144 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] synchronize users...
12:27:57.147 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/api/users/menu/1
12:27:57.306 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/api/users/menu/1 (159ms, unknown-length body)
12:27:57.318 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] users to update: 53
12:27:57.322 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] synchronize taxonomy ranks...
12:27:57.324 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/taxhub/api/taxref/regnewithgroupe2
12:28:05.319 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/taxhub/api/taxref/regnewithgroupe2 (7994ms, 776-byte body)
12:28:05.335 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] taxonomy ranks to update: 41
12:28:05.339 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] synchronize nomenclature types...
12:28:05.343 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/api/nomenclatures/nomenclatures/taxonomy
12:28:06.663 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/api/nomenclatures/nomenclatures/taxonomy (1319ms, unknown-length body)
12:28:06.735 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] nomenclature types to update: 94
12:28:06.746 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] nomenclature to update: 961
12:28:06.747 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] synchronize nomenclature default values...
12:28:06.749 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/api/occtax/defaultNomenclatures
12:28:06.874 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/api/occtax/defaultNomenclatures (125ms, unknown-length body)
12:28:06.892 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] nomenclature default values to update: 17
12:28:06.973 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] synchronize taxa...
12:28:06.975 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=0
12:28:08.398 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=0 (1422ms, 3107413-byte body)
12:28:09.721 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] taxa to update: 10000
12:28:09.722 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=10000
12:28:10.646 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=10000 (923ms, 3398600-byte body)
12:28:12.097 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] taxa to update: 20000
12:28:12.098 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=20000
12:28:14.545 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=20000 (2446ms, 3384678-byte body)
12:28:15.897 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] taxa to update: 30000
12:28:15.898 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=30000
12:28:16.786 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=30000 (887ms, 2714283-byte body)
12:28:18.031 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] taxa to update: 40000
12:28:18.032 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=40000
12:28:18.946 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=40000 (912ms, 2829472-byte body)
12:28:20.100 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] taxa to update: 50000
12:28:20.101 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=50000
12:28:20.991 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=50000 (889ms, 3080240-byte body)
12:28:22.402 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] taxa to update: 60000
12:28:22.403 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=60000
12:28:23.346 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=60000 (942ms, 3113713-byte body)
12:28:24.558 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] taxa to update: 70000
12:28:24.559 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=70000
12:28:25.484 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=70000 (924ms, 3292079-byte body)
12:28:26.731 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] taxa to update: 80000
12:28:26.735 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=80000
12:28:27.940 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=80000 (1204ms, 3041419-byte body)
12:28:29.122 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] taxa to update: 90000
12:28:29.124 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=90000
12:28:30.041 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=90000 (916ms, 3331946-byte body)
12:28:31.317 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] taxa to update: 100000
12:28:31.318 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=100000
12:28:32.260 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=100000 (941ms, 3012408-byte body)
12:28:33.462 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] taxa to update: 110000
12:28:33.466 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=110000
12:28:34.315 INFO: [fr.geonature.datasync.api.ClientKt] <-- 200 https://geonature.lpo-aura.org/taxhub/api/taxref/allnamebylist/100?limit=10000&offset=110000 (847ms, 2225780-byte body)
12:28:35.249 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] taxa to update: 117114
12:28:36.265 DEBUG: [fr.geonature.commons.data.helper.SQLiteSelectQueryBuilder] sql:
  SELECT taxa."_id" AS taxa__id, taxa."name" AS taxa_name, taxa."kingdom" AS taxa_kingdom, taxa."group" AS taxa_group, taxa."name_common" AS taxa_name_common, taxa."description" AS taxa_description, taxa."rank" AS taxa_rank
  FROM taxa taxa
  args: []
12:28:36.269 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] deleting orphaned taxa...
12:28:40.292 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] orphaned taxa deleted: 0
12:28:40.299 INFO: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] synchronize taxa additional data...
12:28:40.302 INFO: [fr.geonature.datasync.api.ClientKt] --> GET https://geonature.lpo-aura.org/api/synthese/color_taxon?code_area_type=M10&limit=10000&offset=0
12:28:48.272 INFO: [fr.geonature.datasync.packageinfo.worker.CheckInputsToSynchronizeWorker] available inputs to synchronize: 0
12:30:40.306 INFO: [fr.geonature.datasync.api.ClientKt] <-- HTTP FAILED: NetworkException(message=timeout)
12:30:40.320 WARN: [fr.geonature.datasync.sync.usecase.DataSyncUseCase] taxa by area synchronization finished with errors
12:30:40.827 WARN: [fr.geonature.datasync.sync.worker.DataSyncWorker] Synchronisation des unités géographiques terminée avec des erreurs
12:30:40.828 INFO: [fr.geonature.datasync.sync.worker.DataSyncWorker] local data synchronization finished with failed tasks in 172594ms
camillemonchicourt commented 6 months ago

Oui en effet, si la fonctionnalité des couleurs de taxons n'est pas activée on ne devrait pas les synchroniser car cela peut être très long sur certaines instances.

Si j'ai bien compris ce soucis est corrigé dans la 2.7.0 à venir, mais à confirmer par Sébastien ?

Sinon tu peux tester et nous dire sur une 2.7.0-rc pour confirmer ?

lpofredc commented 6 months ago

Sinon tu peux tester et nous dire sur une 2.7.0-rc pour confirmer ?

Cette modif n'apparaît que dans la 2.7.0-rc4, qui requiert un GeoNature 2.14.0, non compatible avec ma version donc ;) Merci!

sgrimault commented 6 months ago

Bonjour,

Oui, c'est déjà le cas où si le paramètre sync.code_area_type n'est pas renseigné, la synchronisation des "couleurs" des taxons ne se fait pas.

camillemonchicourt commented 6 months ago

OK donc ça sera corrigé dans la 2.7, super. Pour penser à référencer ces 2 tickets dans la 2.7.0, j'ai commencé son changelog (https://github.com/PnX-SI/gn_mobile_occtax/pull/253)

camillemonchicourt commented 1 month ago

Fait dans la 2.7.0 en ne renseignant pas le paramètre.