Processing/OGR: do not use shapefile as format for temp input files - Githubissues

qgis / QGIS

QGIS is a free, open source, cross platform (lin/win/mac) geographical information system (GIS)

https://qgis.org

GNU General Public License v2.0

10.61k stars 3.01k forks source link

Processing/OGR: do not use shapefile as format for temp input files #29097

Closed qgib closed 5 years ago

qgib commented 5 years ago

Author Name: Tobias Wendorff (Tobias Wendorff) Original Redmine Issue: 21279

Redmine category:processing/core

The algorithms from GDAL's toolbox for vector processing still use shapefiles as intermediate format. This results in corrupted fieldnames, since those are limited in the DBF format. Also, field contents might get corrupted if they're longer than 255 chars.

@ogr2ogr -f "ESRI Shapefile" C:/Users/tobwen/AppData/Local/Temp/processing_5fe343f547494d5e9fc31fe5cc38c868/680b663b94af46ab9516f2acd4141efd/OUTPUT.shp R:/demo.geojson@

Please use GPKG for this.

qgib commented 5 years ago

Author Name: Jürgen Fischer (@jef-n)

tracker_id was changed from 1 to 2

qgib commented 5 years ago

Author Name: Tobias Wendorff (Tobias Wendorff)

Sorry, Jürgen, I strongly disagree that this is a feature request only.

The use of shapefile is an intermediate format is creating unexpected results and is destroying the user's results. Furthermore, ogr2ogr actually is one of the most capable applications out there, which can handle GPKG files. Even if I don't like GPKG that much, it has removed limits, we had with prosperity shapefile over the last decades.

I believe patching this is damn easy, you only need to replace @-F Esri Shapefile@ with @-F GPKG@ and fix filenames from .shp to .gpkg.

Fixing this bug will save hours of processing, energy and might be even some kittens.

qgib commented 5 years ago

Author Name: Jürgen Fischer (@jef-n)

Tobias Wendorff wrote:

I believe patching this is damn easy,

Patch? Pull request?

qgib commented 5 years ago

Author Name: Giovanni Manghi (@gioman)

is this about specifically the OGR based "convert format" tool?

status_id was changed from Open to Feedback

qgib commented 5 years ago

Author Name: Tobias Wendorff (Tobias Wendorff)

Giovanni Manghi wrote:

is this about specifically the OGR based "convert format" tool?

I do not know whether this question was addressed to me or whether I can answer it at all. As far as I can tell, all the vector tools are affected, f.e. Vector geoprocessing -> Buffer vectors

@ogr2ogr C:/Users/tobwen/AppData/Local/Temp/processing_08c1c2be50d4431a82d6d123c3eeec35/ae5cee8d9c484a1986ed5a3db42f1d50/OUTPUT.shp R:/demo.sqlite -dialect sqlite -sql "SELECT ST_Buffer(geometry, 10.0) AS geometry,* FROM 'demo'" -f "ESRI Shapefile"@

qgib commented 5 years ago

Author Name: Giovanni Manghi (@gioman)

status_id was changed from Feedback to Open
assigned_to_id was configured as Giovanni Manghi

qgib commented 5 years ago

Author Name: Giovanni Manghi (@gioman)

subject was changed from GDAL toolbox still use destructive SHP format to Processing/OGR: do not use shapefile as format for temp input files
assigned_to_id removed Giovanni Manghi

qgib commented 5 years ago

Author Name: Tobias Wendorff (Tobias Wendorff)

To prevent oblivion: this is still an issue in QGIS v3.7.0-Master, revision 8c21a9e848

Since latest GDAL (>= 2.5) supports proj6, the use of OGR might be interested to advanced users. Right now, you need to remove all (destroyed) columns, rejoin the original table and write it to a new file.