OSGeo / grass

GRASS GIS - free and open-source geospatial processing engine
https://grass.osgeo.org
Other
849 stars 308 forks source link

[Bug] v.in.ogr: handle columns with unsupported datatype in GRASS DBMI #2628

Closed ninsbl closed 1 year ago

ninsbl commented 2 years ago

Is your feature request related to a problem? Please describe. I have a layer of polygons in an MS SQL database table, that contains a column with Binary datatype (that I assume comes from ArcSDE). ogr2ogr writes that nicely into GeoPackage, meaning SQLite format.

Import of the layer into GRASS GIS however fails unfortunately, as the data type (_Ogrftype: 8) is unsupported.

The error message I am gettng is:

Creating attribute table for layer <schema.TABLE>...
WARNING: Column type (Ogr_ftype: 8) not supported (Ogr_fieldname:
         GDB_GEOMATTR_DATA)
Importing 487 features (OGR layer <schema.TABLE>)...
DBMI-SQLite driver error:
Error in sqlite3_prepare():
table grass_table has 17 columns but 16 values were supplied

DBMI-SQLite driver error:
Error in sqlite3_prepare():
table grass_table has 17 columns but 16 values were supplied

The problematic column is actually created as TEXT, but values of that column are not attempted to transfer.

Also, the error message occurs first after quite long time (minute +) for a dataset with a few hundred small polygons (may be due to the ODBC driver).

Describe the solution you'd like GRASS GIS skips columns of unsupported types, gives a warning but import succeeds. The error message suggests that values of unsupported data types are skipped already, but the unsupported columns(s) are still created.

Describe alternatives you've considered Alternatives would be: a) GRASS GIS supports Binary/BLOB data type b) GRASS GIS allows to select columns to import (not only renaming them) c) GRASS GIS supports -sql option of OGR to create the layer

Additional context With GRASS GIS 8.2 on Ubuntu 20.04 with data from MS SQL and ArcSDE...

ninsbl commented 2 years ago

Please see a reproducible example below:

wget https://github.com/ninsbl/sample_data/raw/main/blobtest.gpkg
v.in.ogr -o --overwrite --verbose input=blobtest.gpkg output=blobtest
ninsbl commented 1 year ago

Hmmm... The fix in https://github.com/OSGeo/grass/pull/2630 seems unfortunately incomplete.

Import does work if the BLOB column is empty, but fails as soon as there is a a value in the BLOB column. I updated the sample data to contain a small SVG fil in the BLOB column for testing (120 KB in total): https://github.com/ninsbl/sample_data/raw/main/blobtest.gpkg

ninsbl commented 1 year ago

BTW: The error message looks as something like this:

ERROR: Cannot insert new row for input layer <LAYER>: insert into
       LAYER values ( 3insert into 3 values ( 3, ....

Inidicating that v.in.ogr repeats part of the insert string for the record with a value in the BLOB column...

ninsbl commented 1 year ago

Should L1420: https://github.com/OSGeo/grass/blob/9ad98566af4142d517df43fd017ef41b5ab2003f/vector/v.in.ogr/main.c#L1418-L1421 actually be replaced with:

                continue;

to skip the column from import entirely?