search

qgis / QGIS

QGIS is a free, open source, cross platform (lin/win/mac) geographical information system (GIS)
https://qgis.org
GNU General Public License v2.0
10.31k stars 2.97k forks source link

Add support for CSV/Delimited text inputs to GDAL tools #31267

Open timworley opened 5 years ago

timworley commented 5 years ago

QGIS version: 3.8.1-Zanzibar QGIS code revision: dcd95cc648 Qt version: 5.11.2 GDAL version: 2.4.1 GEOS version: 3.7.2-CAPI-1.11.0 b55d2125 PROJ version: Rel. 5.2.0, September 15th, 2018 Processing algorithm… Algorithm 'Rasterize (vector to raster)' starting… Input parameters: { 'BURN' : 0, 'DATA_TYPE' : 5, 'EXTENT' : '339398.5533,339783.5924,1641093.171,1641491.101 [EPSG:6445]', 'FIELD' : 'Cond.[mS/m]', 'HEIGHT' : 30, 'INIT' : None, 'INPUT' : 'file:///C:/Users/Tim%20Worley/Desktop/All_Data.csv?type=csv&skipEmptyFields=Yes&detectTypes=yes&xField=Easting&yField=Northing&crs=EPSG:6445&spatialIndex=no&subsetIndex=no&watchFile=no', 'INVERT' : False, 'NODATA' : 0, 'OPTIONS' : '', 'OUTPUT' : 'TEMPORARY_OUTPUT', 'UNITS' : 0, 'WIDTH' : 30 }

GDAL command: gdal_rasterize -l None -a Cond.[mS/m] -ts 30.0 30.0 -a_nodata 0.0 -te 339398.5533 1641093.171 339783.5924 1641491.101 -ot Float32 -of GTiff file:///C:/Users/Tim%20Worley/Desktop/All_Data.csv?type=csv&skipEmptyFields=Yes&detectTypes=yes&xField=Easting&yField=Northing&crs=EPSG:6445&spatialIndex=no&subsetIndex=no&watchFile=no "C:/Users/Tim Worley/AppData/Local/Temp/processing_daf6a36dfb004721be046abfb636b259/c6885c79f18a4e97ae915771acc910a4/OUTPUT.tif" GDAL command output:

FAILURE: No output file specified.

Usage: gdal_rasterize [-b band]* [-i] [-at]

{[-burn value]* | [-a attribute_name] | [-3d]} [-add]

[-l layername]* [-where expression] [-sql select_statement]

[-dialect dialect] [-of format] [-a_srs srs_def] [-to "NAME=VALUE"]*

[-co "NAME=VALUE"] [-a_nodata value] [-init value]

[-te xmin ymin xmax ymax] [-tr xres yres] [-tap] [-ts width height]

[-ot {Byte/Int16/UInt16/UInt32/Int32/Float32/Float64/

CInt16/CInt32/CFloat32/CFloat64}] [-optim {[AUTO]/VECTOR/RASTER}] [-q]

'skipEmptyFields' is not recognized as an internal or external command, operable program or batch file. 'detectTypes' is not recognized as an internal or external command, operable program or batch file. 'xField' is not recognized as an internal or external command, operable program or batch file. 'yField' is not recognized as an internal or external command, operable program or batch file. 'crs' is not recognized as an internal or external command, operable program or batch file. 'spatialIndex' is not recognized as an internal or external command, operable program or batch file. 'subsetIndex' is not recognized as an internal or external command, operable program or batch file. 'watchFile' is not recognized as an internal or external command, operable program or batch file. Execution completed in 0.10 seconds Results: {'OUTPUT': 'C:/Users/Tim ' 'Worley/AppData/Local/Temp/processing_daf6a36dfb004721be046abfb636b259/c6885c79f18a4e97ae915771acc910a4/OUTPUT.tif'} Loading resulting layers The following layers were not correctly generated.
  • C:/Users/Tim Worley/AppData/Local/Temp/processing_daf6a36dfb004721be046abfb636b259/c6885c79f18a4e97ae915771acc910a4/OUTPUT.tif
You can check the 'Log Messages Panel' in QGIS main window to find more information about the execution of the algorithm.
linusmartensson commented 5 years ago

Unintuitively, this seems to be an issue with how the CSV file is sent to gdal_rasterizer, in your case:

file:///C:/Users/Tim%20Worley/Desktop/All_Data.csv?type=csv&skipEmptyFields=Yes&detectTypes=yes&xField=Easting&yField=Northing&crs=EPSG:6445&spatialIndex=no&subsetIndex=no&watchFile=no

The same problem occurs when running the command from a command line and I can reproduce it on my own installation of zanzibar. A similar issue occurs if you have a filename with () in it in the input, and possibly other special characters. Is this a command generation issue? I would expect that & and other special characters may be interpreted if the command is sent directly to a console environment, and it doesn't seem like gdal_rasterizer supports the provided format specification?

As a workaround, converting the csv to a geopackage and rerunning the command seems to work.

gioman commented 5 years ago

The same problem occurs when running the command from a command line and I can reproduce it on my own installation of zanzibar.

probably gdal_rasterize does not support CSV files as inputs. Probably we should not allow QGIS delimited text layers for this tools.

alexbruy commented 5 years ago

There is no proper support for delimited text files in GDAL algorithms. I will try to look into it

titcombj commented 4 years ago

Rasterisation of CSV works for me under 3.0.1. Until I found this page, my workaround has been to load projects up in 3.0.1 every time I want to do this.

bezin commented 4 years ago

I have the very same issue with my xyz data that comes in the format x y z

32351990.00 5717990.00 49.81

My error log is as follows. GDAL error messages are in German unfortunately. It basically says "Command could not be found" every time.

QGIS version: 3.10.4-A Coruña
QGIS code revision: 10bc9f2ff2
Qt version: 5.11.2
GDAL version: 3.0.4
GEOS version: 3.8.1-CAPI-1.13.3
PROJ version: Rel. 6.3.1, February 10th, 2020
Processing algorithm…
Algorithm 'Rasterize (vector to raster)' starting…
Input parameters:
{ 'BURN' : 0, 'DATA_TYPE' : 5, 'EXTENT' : '32334872.25617647,32367117.74382353,5699550.25,5718439.75 [EPSG:5555]', 'EXTRA' : '', 'FIELD' : 'field_3', 'HEIGHT' : 0, 'INIT' : None, 'INPUT' : 'file:///D:/dgm/dgm_oberhausen_output/merged_dgm1_32344_5710_2_nw.xyz?type=csv&delimiter=%20&useHeader=No&skipEmptyFields=Yes&detectTypes=yes&xField=field_1&yField=field_2&crs=EPSG:5555&spatialIndex=no&subsetIndex=no&watchFile=no', 'INVERT' : False, 'NODATA' : 0, 'OPTIONS' : '', 'OUTPUT' : 'C:/Users/benedict.zinke/Downloads/dtm_oberhausen.tif', 'UNITS' : 0, 'WIDTH' : 6000 }

GDAL command:
gdal_rasterize -l None -a field_3 -ts 6000.0 0.0 -a_nodata 0.0 -te 32334872.25617647 5699550.25 32367117.74382353 5718439.75 -ot Float32 -of GTiff file:///D:/dgm/dgm_oberhausen_output/merged_dgm1_32344_5710_2_nw.xyz?type=csv&delimiter=%20&useHeader=No&skipEmptyFields=Yes&detectTypes=yes&xField=field_1&yField=field_2&crs=EPSG:5555&spatialIndex=no&subsetIndex=no&watchFile=no C:/Users/benedict.zinke/Downloads/dtm_oberhausen.tif
GDAL command output:
ERROR 1: Wrong value for -outsize parameter.

Usage: gdal_rasterize [-b band]* [-i] [-at]

{[-burn value]* | [-a attribute_name] | [-3d]} [-add]

[-l layername]* [-where expression] [-sql select_statement]

[-dialect dialect] [-of format] [-a_srs srs_def] [-to "NAME=VALUE"]*

[-co "NAME=VALUE"]* [-a_nodata value] [-init value]*

[-te xmin ymin xmax ymax] [-tr xres yres] [-tap] [-ts width height]

[-ot {Byte/Int16/UInt16/UInt32/Int32/Float32/Float64/

CInt16/CInt32/CFloat32/CFloat64}] [-optim {[AUTO]/VECTOR/RASTER}] [-q]

<src_datasource> <dst_filename>

Der Befehl "delimiter" ist entweder falsch geschrieben oder

konnte nicht gefunden werden.

Der Befehl "useHeader" ist entweder falsch geschrieben oder

konnte nicht gefunden werden.

Der Befehl "skipEmptyFields" ist entweder falsch geschrieben oder

konnte nicht gefunden werden.

Der Befehl "detectTypes" ist entweder falsch geschrieben oder

konnte nicht gefunden werden.

Der Befehl "xField" ist entweder falsch geschrieben oder

konnte nicht gefunden werden.

Der Befehl "yField" ist entweder falsch geschrieben oder

konnte nicht gefunden werden.

Der Befehl "crs" ist entweder falsch geschrieben oder

konnte nicht gefunden werden.

Der Befehl "spatialIndex" ist entweder falsch geschrieben oder

konnte nicht gefunden werden.

Der Befehl "subsetIndex" ist entweder falsch geschrieben oder

konnte nicht gefunden werden.

Der Befehl "watchFile" ist entweder falsch geschrieben oder

konnte nicht gefunden werden.

Execution completed in 0.14 seconds

I am absolutely new to GIS, but I am happy to provide more input. Would also be happy about any workaround :-)