Samples do not show on map when one of them have missing coordinates

[jiho]

When arriving on the main page, the map stays empty. When selecting one project it fills with the points of that one. When erasing the selection with the button "Display all" it keeps only the previous points on the map and still does not show the full set of points.

[grololo06]

I dumped the fetched JSON and it's not valid, I guess the problem is there: {"id": 34557, "lat": NaN, "long": NaN, "visibility": "YY"}, {"id": 34558, "lat": NaN, "long": NaN, "visibility": "YY"} Some samples have no valid coordinates, so the parsing fails.

[grololo06]

Finding which project(s) contain these samples:

root@ecotaxa:/home/ecotaxa# psql -U postgres -d ecopart -h localhost -p 5435
L'utilisation du paginateur est désactivé.
Chronométrage activé.
psql (14.3 (Ubuntu 14.3-1.pgdg18.04+1), serveur 13.7 (Ubuntu 13.7-1.pgdg18.04+1))
Connexion SSL (protocole : TLSv1.3, chiffrement : TLS_AES_256_GCM_SHA384, bits : 256, compression : désactivé)
Saisissez « help » pour l'aide.

ecopart=# select * from part_samples where psampleid in (34557, 34558);
 psampleid | pprojid |       profileid       |                   filename                   | sampleid | latitude | longitude | organizedbydeepth | histobrutavailable | qualitytaxo | qualitypart | daterecalculhistotaxo | winddir | winspeed | seastate | nebuloussness | comment | stationid | firstimage | lastimg | lastimgused | bottomdepth | yoyo |     sampledate      | op_sample_name | op_sample_email | ctd_desc | ctd_origfilename | ctd_import_name | ctd_import_email | ctd_import_datetime | ctd_status | instrumsn | acq_aa | acq_exp | acq_volimage | acq_depthoffset | acq_pixel | acq_shutterspeed | acq_smzoo | acq_exposure | acq_gain | acq_filedescription | acq_eraseborder | acq_tasktype | acq_threshold | acq_choice | acq_disktype | acq_smbase | acq_ratio | acq_descent_filter | acq_presure_gain | acq_xsize | acq_ysize | acq_barcode | proc_datetime | proc_gamma | proc_soft | lisst_zscat_filename | lisst_kernel | lisst_year | txt_data01 | txt_data02 | txt_data03 | txt_data04 | txt_data05 | txt_data06 | txt_data07 | txt_data08 | txt_data09 | txt_data10 | imp_descent_filtered_row | imp_removed_empty_slice | proc_process_ratio | integrationtime 
-----------+---------+-----------------------+----------------------------------------------+----------+----------+-----------+-------------------+--------------------+-------------+-------------+-----------------------+---------+----------+----------+---------------+---------+-----------+------------+---------+-------------+-------------+------+---------------------+----------------+-----------------+----------+------------------+-----------------+------------------+---------------------+------------+-----------+--------+---------+--------------+-----------------+-----------+------------------+-----------+--------------+----------+---------------------+-----------------+--------------+---------------+------------+--------------+------------+-----------+--------------------+------------------+-----------+-----------+-------------+---------------+------------+-----------+----------------------+--------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+--------------------------+-------------------------+--------------------+-----------------
     34557 |     279 | lovuse002b-063-03-ASC | lovuse002b-063-03-ASC_000108LP_DEPTH_LPM.txt |          |      NaN |       NaN | t                 | t                  |             |             |                       |         |          |          |               |         |           |            |         |             |             |      | 2022-05-12 15:36:54 |                |                 |          |                  |                 |                  |                     |            | 000108LP  |   2342 |   1.136 |          0.6 |             0.6 |        73 |              268 |           |              |        6 |                     |                 |              |            20 |            |              |            |           |                    |                  |           |           |             |               |            |           |                      |              |            |            |            |            |            |            |            |            |            |            |            |                          |                         |                    |                
     34558 |     279 | lovuse002b-063-03-PAR | lovuse002b-063-03-PAR_000108LP_TIME_LPM.txt  |          |      NaN |       NaN | f                 | t                  |             |             |                       |         |          |          |               |         |           |            |         |             |             |      | 2022-05-07 19:52:20 |                |                 |          |                  |                 |                  |                     |            | 000108LP  |   2342 |   1.136 |          0.6 |             0.6 |        73 |              268 |           |              |        6 |                     |                 |              |            20 |            |              |            |           |                    |                  |           |           |             |               |            |           |                      |              |            |            |            |            |            |            |            |            |            |            |            |                          |                         |                    |                
(2 lignes)

=> prj 279

[grololo06]

Random thoughts:

The samples can also be viewed individually with a URL: https://ecopart.obs-vlfr.fr/sampleedit/34557 and https://ecopart.obs-vlfr.fr/sampleedit/34558

The DB reveals that _only_these 2 samples have an invalid lat/lon:

ecopart=# select * from part_samples where latitude = 'NaN' or longitude = 'NaN';
(2 lignes)

Samples imported later in same project are OK, e.g. https://ecopart.obs-vlfr.fr/sampleedit/34773

The samples were imported nightly, so the job trace contains some output on them:

root@ecotaxa:/home/ecopart# vi DailyTask.out
vendredi 13 mai 2022, 01:15:01 (UTC+0000)
...
pollserver for project 279 : uvp6_sn000108lp_2020_07_float_remote
Processing sample  lovuse002b-063-03-ASC
Processing sample  lovuse002b-063-03-PAR
uvp6remote Sample 34557 Metadata processed, Détailled histogram in progress
Try to import CTD
False
uvp6remote Sample 34558 Metadata processed, Détailled histogram in progress
Try to import CTD
False

So the problem looks like invalid upstream data, it's unclear what should be done in such case as e.g. in this case the imported data is not present anymore in the source (see project import URL).

[picheral]

I checked the data files on the server : http://www.oao.obs-vlfr.fr/bioargo/PHP/lovuse002b/UVP6/

The lat and long are documented by NA. The samples should thus not be imported to avoid any problem in EcoPART => modification of the code (no LAT/LON => file skipped). The problem is that no one will know that the sample is skipped....

[picheral]

Fermée et remplacée par #58

[grololo06]

OK le problème d'affichage n'est qu'une conséquence et #58 indique clairement que ces samples ne devraient pas être là, mais pour rétablir l'affichage sur la prod', on fait quoi? On supprime les samples? On envoie tout ça au pôle sud? :)

[grololo06]

Status: Il y a une conversation en cours avec le propriétaire des données pour éviter le problème, e.g. en filtrant celles-ci avant l'import et en attendant que #58 soit implémenté. Note: Il est aussi possible de filtrer les 'NaN' lors de l'affichage ce qui rétablirait celui-ci en laissant les data sans GPS (dans py/part_app/views/part_main.py)

[picheral]

Je pense qu'autoriser l'import et sécuriser l'affichage est la meilleure solution. Il faudra vérifier que l'export se fait normalement avec des NUL en place des LAT-LON manquantes.