search

qgis / QGIS

QGIS is a free, open source, cross platform (lin/win/mac) geographical information system (GIS)
https://qgis.org
GNU General Public License v2.0
10.52k stars 2.99k forks source link

Processing/SAGA: translate inputs to GPKG instead of SHP #47100

Closed lucalanteri closed 1 year ago

lucalanteri commented 2 years ago

What is the bug or the crash?

Some algorithms don't work when the starting layer is a gpkg with a long name field. The problem seems to be that the starting layer is exported in shapefile format, the long name is truncated, but the command still use the original layer name.

Error: executing tool [Thin Plate Spline (TIN)] 2022-01-31T20:57:49 INFO SAGA execution commands grid_spline "Thin Plate Spline (TIN)" -TARGET_DEFINITION 0 -SHAPES "/tmp/processing_MyuIjo/1836dbcd0d794855854e7d62fbcb2995/SHAPES.shp" -FIELD "campoconnomelungo" -REGULARISATION 0.0001 -LEVEL 0 -FRAME true -TARGET_USER_SIZE 100.0 -TARGET_USER_FITS 0 -TARGET_OUT_GRID "/tmp/processing_MyuIjo/ebf9df1e7f3645338ab182ed63bdf427/TARGET_OUT_GRID.sdat" 2022-01-31T20:57:50 INFO SAGA execution console output

Steps to reproduce the issue

Versions

QGIS version | 3.23.0-Master

Qt version | 5.12.8 Python version | 3.8.10 GDAL/OGR version | 3.0.4 PROJ version | 6.3.1 EPSG Registry database version | v9.8.6 (2020-01-22) Compiled against GEOS | 3.8.0-CAPI-1.13.1 | Running against GEOS | 3.8.0-CAPI-1.13.1 SQLite version | 3.31.1 PostgreSQL client version | 14.1 (Ubuntu 14.1-2.pgdg20.04+1) SpatiaLite version | 4.3.0a QWT version | 6.1.4 QScintilla2 version | 2.11.2 OS version | Ubuntu 20.04.3 LTS Active Python plugins qtiles | 1.7.1 db-style-manager | 0.8 OSMDownloader | 1.0.3 QuickOSM | 2.0.0 quick_map_services | 0.19.27 sagaprovider | 2.12.99 db_manager | 0.1.20 processing | 2.12.99 grassprovider | 2.12.99

Supported QGIS version

New profile

Additional context

sampledata.gpkg.zip

gioman commented 2 years ago

@lucalanteri does SAGA tools with the "-SHAPES" parameter accept anything other than Shapefiles?

lucalanteri commented 2 years ago

I think SAGA accepts only Shapefile as vector format.

gioman commented 2 years ago

I think SAGA accepts only Shapefile as vector format.

@lucalanteri if is that way, what other options QGIS has (if not translating the GPKG input to SHP)?

lucalanteri commented 2 years ago

I think it could be try to use the filedname trimmed to 10 char. Eg: parameter -FIELD "campoconno" instead of the original -FIELD "campoconnomelungo".

The only problem is that if we have 2 fields that start with the same 10 char, qgis convert the name in campocon_1 and campocon_2.

nicogodet commented 2 years ago

I don't get it Shapefile don't support field name longer than 10 char...

lucalanteri commented 2 years ago

@nicogodet a solution could be to use the parameter "-FIELD" according to the new field name (trimmed by shapefile conversion) instead of the old long name in gpkg.

esnyder-rve commented 2 years ago

SAGA does have the ability to open & load GPKG data.

Screenshot 2022-02-02 163017

It opens the whole GPKG though, so there might be an added step needed to move the data to a temp GPKG that just contains that one layer.

gioman commented 2 years ago

SAGA does have the ability to open & load GPKG data.

@esnyder-rve right. But for our use what it matters is how it works from the CLI. From some test I have made it seems that (for example) the SHAPES parameter accepts also GPKGs. When there are multiple layers in a GPKG it seems to go for the first one (as reported by ogrinfo), and I don't see a parameter in SAGA tools to specify what layer to use.

esnyder-rve commented 2 years ago

So in that case @gioman,

so there might be an added step needed to move the data to a temp GPKG that just contains that one layer.

However, this might not be desirable for large datasets.

gioman commented 2 years ago

However, this might not be desirable for large datasets.

@esnyder-rve yes, but the advantage (of the translation to SHP) is that users can process any type of vector layers they have loaded in the project (even from databases for example). Is a trade off, of course, but seems to me that is a big advantage and user just need to be conscious about this translation on the fly (arguably this ticket could become a request to ask translate to GPKG instead of SHP so to avoid column names truncation).

esnyder-rve commented 2 years ago

I think that since a conversion would still be required, converting to a GPKG would be a better choice in my opinion. I also agree that the user should be told about this copy.

alexbruy commented 1 year ago

SAGA Provider was not maintained and now remowed from QGIS (see #50834). Users are encouraged to use the 3rd-party plugins instead.