GPKG to Spatialite: add validity check on cross_section_table inputs

[Patrick-Nijman]

• [x] https://github.com/nens/servicedesk-issue-only/issues/96

If the geopackage cross_section_table input cannot be correctly parsed to width / height series in the spatialite, data can be lost. Some examples:

1. empty cross section table (NULL)
2. cross section tables with extra blank line
3. cross section table with space after comma
4. cross section table with a . in stead of , between width and height

Suggested solution:

• [x] Make conversion algorithm more robust, so that cases like 1, 2, and 3 are handled correctly
• [x] Check validity of this input before starting export. If not valid, show popup message to user.
• [x] Implement #64

"Saving to Spatialite failed. The following features have cross sections with incorrect table inputs: Cross section location {ids} Culvert {ids} Orifice {ids} Pipe {ids} Weir {ids}"

[caspervdw]

Patrick: Not important to Lukasz, basically a description why Casper can't solve this in the model checker. I thought this needed fixing in the modelchecker. But it's the conversion from GPckg to Sqlite

Ik heb het lijstje afgelopen en ik begrijp eigenlijk alle punten niet. Ik vermoed dat dit over records in de v2_cross_section_definition table met een tabulated shape gaat (dus: shape = 5 of 6).

-empty cross section table (NULL)

Er is al een check op NULL waardes in de width kolom (error code 81) of height kolom (error code 82)

-cross section tables with extra blank line

Wat bedoel je precies? een newline aan het einde van width of height veld is valid input en geeft geen errors in gridbuilder.

-cross section table with space after comma

Er zitten geen komma's in de input

-cross section table with a . in stead of , between with and height

Width en height zitten in aparte velden; daar zit geen . of , tussen.

[caspervdw]

Discussed with @Patrick-Nijman -- issue is in the plugin (conversion gpkg -> sqlite)

[ldebek]

Changes available on branch: https://github.com/nens/threedi-schematisation-editor/tree/cross_section_table_validators

[leendertvanwolfswinkel]

I tested it. Test results:

• Performance for 'Save to spatialite' has deteriorated quiet a bit: QGIS freezes for a while and the user does not know what is going on. Please include the validation in the progress bar.
• When I have no active edit sessions and use Save to spatialite, I get a lot of these popups: Not only for Manhole, but also for Weir, Culvert, Orifice, Pipe, Manhole (again), Manhole (again), Manhole (again). And than it works. Please commit all required edits without the popups.
• If I add a blank line at the end of my input, I get Saving to Spatialite failed. The following features have cross sections with incorrect table inputs. Please auto-fix this kind of input (just remove all spaces and blank lines before trying to parse the input?).
• If saving to Spatialite fails, all layers remain in edit mode. Please end all edit sessions (without popups) after failing Save to Spatialite.
• If the input is 0,2.5\n1.5,2.5\n1.5,0, save to spatialite fails. Please make sure input (without spaces) and 0, 2.5\n1.5, 2.5\n1.5, 0 (with spaces) are accepted and correctly parsed. 0, 2,5\n1,5, 2,5\n1,5, 0 (comma's as decimal separators) does not have to be accepted
• If the table input is empty, at the end of the Save to Spatialite process, I get an error (see below). Please make sure empty input is also correctly handled (should result in empty v2_cross_section_definition.width and v2_cross_section_definition.height fields

ValueError: not enough values to unpack (expected 2, got 1) 
Traceback (most recent call last):
  File "C:\Users/leendert.vanwolfswin/AppData/Roaming/QGIS/QGIS3\profiles\Release candidates June 2022/python/plugins\threedi_schematisation_editor\__init__.py", line 195, in save_to_default
    self.save_to_spatialite(pick_destination=False)
  File "C:\Users/leendert.vanwolfswin/AppData/Roaming/QGIS/QGIS3\profiles\Release candidates June 2022/python/plugins\threedi_schematisation_editor\__init__.py", line 243, in save_to_spatialite
    converter.export_all_model_data()
  File "C:\Users/leendert.vanwolfswin/AppData/Roaming/QGIS/QGIS3\profiles\Release candidates June 2022/python/plugins\threedi_schematisation_editor\conversion.py", line 706, in export_all_model_data
    self.export_cross_section_definition_data()
  File "C:\Users/leendert.vanwolfswin/AppData/Roaming/QGIS/QGIS3\profiles\Release candidates June 2022/python/plugins\threedi_schematisation_editor\conversion.py", line 611, in export_cross_section_definition_data
    height_values, width_values = list(zip(*parsed_table))
ValueError: not enough values to unpack (expected 2, got 1)

[leendertvanwolfswinkel]

Tested again, all previously mentioned issues have been solved now, except for the last two:

• [x] If the input is 0,2.5\n1.5,2.5\n1.5,0, save to spatialite fails. Please make sure input (without spaces) and 0, 2.5\n1.5, 2.5\n1.5, 0 (with spaces) are accepted and correctly parsed. 0, 2,5\n1,5, 2,5\n1,5, 0 (comma's as decimal separators) does not have to be accepted
• [x] If the table input is empty, at the end of the Save to Spatialite process, I get an error (see below). Please make sure empty input is also correctly handled (should result in empty v2_cross_section_definition.width and v2_cross_section_definition.height fields

[ldebek]

@leendertvanwolfswinkel Those 2 issues should be solved now.

[leendertvanwolfswinkel]

@ldebek Both issues are solved, but now I get several "Do you want to save changes to the layer Manhole" popups again (I think four times, between data validation and saving to spatialite.

This also happens if I toggle editing on for the Pipe layer and then off again, even when I haven't edited anything.

[leendertvanwolfswinkel]

More info to reproduce the issue:

• Load the following spatialite: Hekerbeek na maatregelen Gebiedsbreed T25.zip
• Use Identify tool to open attribute form for Pipe with ID 18891
• Start editing
• Stop editing

[ldebek]

@leendertvanwolfswinkel Issue reported in comment above is not related with validation. It seems that there is an issue with handling NULL values in the custom forms comboboxes. Best to move it to the separate ticket.