andrecastro0o commented 2 months ago

implement a error handling strategy in the export script that ignores empty rows, but writes them down in log files and NetCDF file (@agoncalvesdeol make this a github issue)

The example DB where the 1st 5 rows (ids 1-5) are empty can be found in https://ruisdael.citg.tudelft.nl/parsivel/disdrodl_PAR004_emptyrows.db

The export error:

/usr/local/src/disdrodl/venv/bin/python /usr/local/src/disdrodl/export_disdrodlDB2NC.py -d 2024-08-07 -c /usr/local/src/disdrodl/configs_netcdf/config_PAR_004_WESTMAAS.yml

self.telegram_lines is EMPTY. self.telegram_lines: 
self.telegram_lines is EMPTY. self.telegram_lines: 
self.telegram_lines is EMPTY. self.telegram_lines: 
self.telegram_lines is EMPTY. self.telegram_lines: 
Traceback (most recent call last):
  File "/usr/local/src/disdrodl/export_disdrodlDB2NC.py", line 164, in <module>
    main(get_arguments())
  File "/usr/local/src/disdrodl/export_disdrodlDB2NC.py", line 122, in main
    telegram_instance.parse_telegram_row()
  File "/usr/local/src/disdrodl/modules/telegram.py", line 209, in parse_telegram_row
    self.str2list(field='90', separator=',')
  File "/usr/local/src/disdrodl/modules/telegram.py", line 143, in str2list
    str_val = self.telegram_data[field]
              ~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: '90'

andrecastro0o commented 2 months ago

Tests

start by writing a test for it.

use https://ruisdael.citg.tudelft.nl/parsivel/disdrodl_PAR004_emptyrows.db as a test case

Approach

there are 2 ways to approach this:

1: by filtering out in the SQL query the entries that have empty telegrams ie sqlite3 disdrodl_PAR004_emptyrows.db "SELECT * FROM disdroDL WHERE id < 7 AND LENGTH(telegram) > 100"
2: by using an if statement in the export loop, that inserts them into the NetCDF

I prefer option on, but it is important to notify the user what were the timestamps of the missing telegrams, probably in NetCDF comment: key

The reason for performing the LENGTH(telegram) > 100" is that we can have entries in the telegram column which are not telegram, as the following example query to of disdrodl_PAR004_emptyrows.db shows

sqlite3 disdrodl_PAR004_emptyrows.db "SELECT * FROM disdroDL WHERE id < 7 AND telegram != ''"

5|1724761500.50004|2024-08-27T12:25:00.500039+00:00|PAR004|VERSION:2.11.6; BUILD:2112151
6|1724761560.98882|2024-08-27T12:26:00.988821+00:00|PAR004|01:0000.000; 02:0000.00; 03:00; 04:00; 05:NP; 06:C; 07:-9.999; 08:20000; 09:00044; 10:21033; 11:00000; 12:048; 13:450904; 14:2.11.6; 15:2.11.1; 16:0.50; 17:24.3; 18:0; 19:None; 20:13; 21:27.08.2024; 22:WESTMAAS; 23:004; 24:0000.00; 25:000; 26:035; 27:034; 28:030; 29:000.559; 30:00.000; 31:0000.0; 32:0000.00; 34:0000.00; 35:0000.00; 40:20000; 41:20000; 50:00000000; 51:000140; 90:-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999,-9.999; 91:00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000,00.000; 93:000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000; 94:0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000,0000; 95:0.00,0.00,0.00,0.00,0.00,0.00,0.00; 96:0000000,0000000,0000000,0000000,0000000,0000000,0000000

andrecastro0o commented 2 months ago

merge to 0.3.3 branch

ruisdael-observatory / disdroDL

handle empty DB rows in export script #28

Tests

Approach