search

OSGeo / gdal

GDAL is an open source MIT licensed translator library for raster and vector geospatial data formats.
https://gdal.org
Other
4.87k stars 2.54k forks source link

gdalrasterize is returning a 3x3 matrix when options request a 2x3 #10981

Closed joa-quim closed 2 weeks ago

joa-quim commented 3 weeks ago

What is the bug?

Somehow this commit

SHA-1: 462285a9e8101e630bb52a0c7651ce7247d06de7

* Merge pull request #10742 from rouault/netcdf_identify_hdf4_hdf5

netCDF driver: simplify identification logic by just checking runtime  availability of HDF4/HDF5 drivers

breaks the GMT Julia wrapper when it calls a gdalrasterize wrapper function.

Don't know how best to help but before this commit these options ["-of", "MEM", "-ts", "3", "2", "-burn", "1", "-ot", "Byte"] made it return a 2x3 matrix, as required, and after that commit it returns a 3x3 matrix.

In case it helps, the code in question is in https://github.com/GenericMappingTools/GMT.jl/blob/master/src/rasterpolygonfuns.jl#L254 and the input (shapes[4]) is this matrix

1.6  2.6
1.6  4.4
4.4  4.4
4.4  2.6
1.6  2.6

Steps to reproduce the issue

Must run the julia code to reproduce it.

using GMT

D = mat2ds([1.6 2.6; 1.6 4.4; 4.4 4.4; 4.4 2.6; 1.6 2.6]); im = mat2img(UInt8.(GMT.magic(9)));
colorzones!(D, img=im)

Versions and provenance

Windows 11 and MSVC build

Additional context

No response

rouault commented 3 weeks ago

Must run the julia code to reproduce it.

I'm not set up for Julia things. Any chance you can reproduce that with gdal_rasterize command line? At the very least if you can extract the source dataset as a standalone file that could help. Is it a netCDF or HDF5 dataset? Otherwise I don't see the relationship with commit 462285a9e8101e630bb52a0c7651ce7247d06de7

joa-quim commented 2 weeks ago

Note that, the https://github.com/OSGeo/gdal/commit/462285a9e8101e630bb52a0c7651ce7247d06de7 also changed gdal_rasterize_bin|lib.cpp

Ok here is what I found. If I convert my input test data [1.6 2.6; 1.6 4.4; 4.4 4.4; 4.4 2.6; 1.6 2.6] to .gpkg

D.gpkg.zip

Than with the previous commit (where it sill works), I can do

gdal_rasterize -ts 3 2 -burn 1 -ot Byte D.gpkg D.tiff
0...10...20...30...40...50...60...70...80...90...100 - done.

C:\v>gdalinfo D.tiff
Driver: GTiff/GeoTIFF
Files: D.tiff
Size is 3, 2
Origin = (1.600000000000000,4.400000000000000)
Pixel Size = (0.933333333333333,-0.900000000000000)
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (   1.6000000,   4.4000000)
Lower Left  (   1.6000000,   2.6000000)
Upper Right (   4.4000000,   4.4000000)
Lower Right (   4.4000000,   2.6000000)
Center      (   3.0000000,   3.5000000)
Band 1 Block=3x2 Type=Byte, ColorInterp=Gray

but with https://github.com/OSGeo/gdal/commit/462285a9e8101e630bb52a0c7651ce7247d06de7 it refuses to run the same command.

gdal_rasterize -ts 3 2 -burn 1 -ot Byte D.gpkg D.tiff
Usage: gdal_rasterize [--help] [--help-general]
       [-b <band>]... [-i] [-at]
       [-oo <NAME>=<VALUE>]...
       {[-burn <value>]... | [-a <attribute_name>] | [-3d]} [-add]
       [-l <layername>]... [-where <expression>] [-sql <select_statement>|@<filename>]
       [-dialect <dialect>] [-of <format>] [-a_srs <srs_def>] [-to <NAME>=<VALUE>]...
       [-co <NAME>=<VALUE>]... [-a_nodata <value>] [-init <value>]...
       [-te <xmin> <ymin> <xmax> <ymax>] [-tr <xres> <yres>] [-tap] [-ts <width> <height>]
       [-ot {Byte/Int8/Int16/UInt16/UInt32/Int32/UInt64/Int64/Float32/Float64/
             CInt16/CInt32/CFloat32/CFloat64}] [-optim {AUTO|VECTOR|RASTER}] [-q]
       <src_datasource> <dst_filename>

FAILURE: No output file specified.
jratike80 commented 2 weeks ago

I can partly re-produce on Windows (OSGeo4W). Command works for me but the output is 3x3.

gdalinfo --version
GDAL 3.10.0dev-4e28921731, released 2024/10/07

gdal_rasterize -ts 3 2 -burn 1 -ot Byte D.gpkg D.tiff
0...10...20...30...40...50...60...70...80...90...100 - done.

gdalinfo D.tiff
Driver: GTiff/GeoTIFF
Files: D.tiff
Size is 3, 3
...

Debug shows that 3x3 is intentional:

...
GDAL: GDALOpen(D.gpkg, this=0000017412131020) succeeds as GPKG.
GDAL: Using GTiff driver
GDAL: GDALDriver::Create(GTiff,D.tiff,3,3,1,Byte,0000000000000000)
GPKG: ResetStatement(SELECT m."fid", m."geom" FROM "layer1" m)
...

And then gdal_rasterize -ts 2 3... makes 2x2 GDAL: GDALDriver::Create(GTiff,E.tiff,2,2,1,Byte,0000000000000000)

joa-quim commented 2 weeks ago

And it works up to

gdalinfo --version
GDAL 3.10.0dev-e78d91e529, released 2024/09/06

but not after.

rouault commented 2 weeks ago

Note that, the 462285a also changed gdal_rasterize_bin|lib.cpp

@joa-quim No it doesn't. This touches only the netCDF driver only. You must be making a confusion with another commit

joa-quim commented 2 weeks ago

Well, I'm just saying that because my Turtoise GUI shows me this

image

rouault commented 2 weeks ago

@joa-quim Thanks for reporting. will be fixed per https://github.com/OSGeo/gdal/pull/10983 (Actually the offending commit was 8d9ceabc12c . What you reported was probably the commit at which you had updated, not the one that caused the issue. anyway that's just nit-picking)

joa-quim commented 2 weeks ago

Hmm, the problem is solved as now it return the 2x3 matrix that the Julia test expected, but oddly, on CLI, this still fails

C:\v>gdal_rasterize -ts 3 2 -burn 1 -ot Byte D.gpkg D.tiff
Usage: gdal_rasterize [--help] [--help-general]
       [-b <band>]... [-i] [-at]
       [-oo <NAME>=<VALUE>]...
       {[-burn <value>]... | [-a <attribute_name>] | [-3d]} [-add]
       [-l <layername>]... [-where <expression>] [-sql <select_statement>|@<filename>]
       [-dialect <dialect>] [-of <format>] [-a_srs <srs_def>] [-to <NAME>=<VALUE>]...
       [-co <NAME>=<VALUE>]... [-a_nodata <value>] [-init <value>]...
       [-te <xmin> <ymin> <xmax> <ymax>] [-tr <xres> <yres>] [-tap] [-ts <width> <height>]
       [-ot {Byte/Int8/Int16/UInt16/UInt32/Int32/UInt64/Int64/Float32/Float64/
             CInt16/CInt32/CFloat32/CFloat64}] [-optim {AUTO|VECTOR|RASTER}] [-q]
       <src_datasource> <dst_filename>

FAILURE: No output file specified.
rouault commented 2 weeks ago

Hmm, the problem is solved as now it return the 2x3 matrix that the Julia test expected, but oddly, on CLI, this still fails

are you fully updated to latest GDAL master ? In particular check that you have this commit ee60a6fb7061a7f7d1ea0d57af571bb962c5b7cf

joa-quim commented 2 weeks ago

Yes, I'm on master. Triple checked. I have to be since the gdal_rasterize: properly honour -ts height (master only) was well applied and working. When I did a git checkout to that commit and rebuilt the error persisted too.

jratike80 commented 2 weeks ago

Do you mean the FAILURE: No output file specified. error? I was wondering where did you get it. I didn't, and the error means that somehow the command line arguments are not parsed correctly. The command did work for me by copy-paste, so I guess that there are no invisible/odd characters in the command. I would still rewrite the command carefully with a keyboard.

joa-quim commented 2 weeks ago

Yes, the online synopsis and that error. Note, for me this error started to occur in the same commit (did not check this thoroughly) that broke the file size. I mean, it worked before, but no longer.

jratike80 commented 2 weeks ago

Odd. With OSGeo4W

gdalinfo --version
GDAL 3.10.0dev-4e28921731, released 2024/10/07
gdal_rasterize -ts 3 2 -burn 1 -ot Byte D.gpkg F.tiff
0...10...20...30...40...50...60...70...80...90...100 - done.
joa-quim commented 2 weeks ago

Indeed, very strange. It does work with a somewhat older OSGeo4W build

c:\v>gdalinfo --version
GDAL 3.9.1, released 2024/06/22
jratike80 commented 2 weeks ago

Are you sure that the new version that you have is really new? The argument parser that was added lately gives different and more readable errors, like

gdal_rasterize -ts 3 2 -burn 1 -ot Byte D.gpkg
ERROR 1: dst_filename: 1 argument(s) expected. 0 provided.
Usage: gdal_rasterize [--help] [--long-usage] [--help-general]
...
joa-quim commented 2 weeks ago

Yes, I get the same parsing message :)

C:\v>gdal_rasterize -ts 3 2 -burn 1 -ot Byte D.dpkg
ERROR 1: dst_filename: 1 argument(s) expected. 0 provided.
Usage: gdal_rasterize [--help] [--help-general]
C:\v>gdalinfo --version
GDAL 3.10.0dev-84ce7fd3ce, released 2024/10/10
jratike80 commented 2 weeks ago

And what with a proper command? gdal_rasterize -ts 3 2 -burn 1 -ot Byte D.dpkg out.tif

rouault commented 2 weeks ago

Yes, I get the same parsing message :)

actually, not, yours lack the "[--long-usage]". There's something particular in your setup that we are missing. Aren't you mixing the "gdal_rasterize" binary from one build with the libgdal.dll from another build, or something like that?

joa-quim commented 2 weeks ago

actually, not, yours lack the "[--long-usage]".

Because I trimmed it

gdal_rasterize -ts 3 2 -burn 1 -ot Byte D.dpkg
ERROR 1: dst_filename: 1 argument(s) expected. 0 provided.
Usage: gdal_rasterize [--help] [--help-general]
       [-b <band>]... [-i] [-at]
       [-oo <NAME>=<VALUE>]...
       {[-burn <value>]... | [-a <attribute_name>] | [-3d]} [-add]
       [-l <layername>]... [-where <expression>] [-sql <select_statement>|@<filename>]
       [-dialect <dialect>] [-of <format>] [-a_srs <srs_def>] [-to <NAME>=<VALUE>]...
       [-co <NAME>=<VALUE>]... [-a_nodata <value>] [-init <value>]...
       [-te <xmin> <ymin> <xmax> <ymax>] [-tr <xres> <yres>] [-tap] [-ts <width> <height>]
       [-ot {Byte/Int8/Int16/UInt16/UInt32/Int32/UInt64/Int64/Float32/Float64/
             CInt16/CInt32/CFloat32/CFloat64}] [-optim {AUTO|VECTOR|RASTER}] [-q]
       <src_datasource> <dst_filename>
joa-quim commented 2 weeks ago

And what with a proper command? gdal_rasterize -ts 3 2 -burn 1 -ot Byte D.dpkg out.tif

That's what I reported several times above. It fails and prints the synopsis

jratike80 commented 2 weeks ago

Now have a cup of coffee and open your eyes. This line is different for me (thank Even for noticing):

Usage: gdal_rasterize [--help] [--long-usage] [--help-general]

gdal_rasterize -ts 3 2 -burn 1 -ot Byte D.gpkg
ERROR 1: dst_filename: 1 argument(s) expected. 0 provided.
Usage: gdal_rasterize [--help] [--long-usage] [--help-general]
                      [-b <band>]... [-i] [-at]
                      [[-burn <value>]...|[-a <attribute_name>]|[-3d]]
                      [-add]
                      [[-l <layer_name>]...|[-sql <sql_statement>]]
                      [-where <expression>] [-dialect <sql_dialect>] [-a_nodata <value>] [-init <value>]...
                      [-a_srs <srs_def>] [-to <NAME>=<VALUE>]... [-te <xmin> <ymin> <xmax> <ymax>]
                      [[-tr <xres> <yres>]|[-ts <width> <height>]]
                      [-tap] [-optim AUTO|VECTOR|RASTER] [-co <NAME>=<VALUE>]...
                      [-ot Byte|Int8|[U]Int{16|32|64}|CInt{16|32}|[C]Float{32|64}] [-of <output_format>]
                      [-oo <NAME>=<VALUE>]... [--quiet]
                      <src_datasource> <dst_filename>
rouault commented 2 weeks ago

Because I trimmed it

no! That's just not the help message of gdal_rasterize from master. This is the one from GDAL 3.9 (or from an older GDAL master prior to the conversion of gdal_rasterize to using argparser). You're definitely mixing GDAL versions here

joa-quim commented 2 weeks ago

OK, I'll investigate more. I should not be the gdal.dll because, being aware of these problems, I maintain only one of it (with a unique name) that I symlink from a unique location. But yes, something fishy is going one here. Weird that it worked for older master versions.

joa-quim commented 2 weeks ago

Shouldn't it be possible to download the binaries that are created for the CI tests?

I found the why for the difference between the help messages. While I'm very careful with the dll because of the conflicts. I'm more loose with the binaries and indeed I was using an older gdal_rasterize binary. I now get

gdal_rasterize -ts 3 2 -burn 1 -ot Byte D.dpkg
ERROR 1: dst_filename: 1 argument(s) expected. 0 provided.
Usage: gdal_rasterize [--help] [--long-usage] [--help-general]
                      [-b <band>]... [-i] [-at]
                      [[-burn <value>]...|[-a <attribute_name>]|[-3d]]
                      [-add]
                      [[-l <layer_name>]...|[-sql <sql_statement>]]
                      [-where <expression>] [-dialect <sql_dialect>] [-a_nodata <value>] [-init <value>]...
                      [-a_srs <srs_def>] [-to <NAME>=<VALUE>]... [-te <xmin> <ymin> <xmax> <ymax>]
                      [[-tr <xres> <yres>]|[-ts <width> <height>]]
                      [-tap] [-optim AUTO|VECTOR|RASTER] [-co <NAME>=<VALUE>]...
                      [-ot Byte|Int8|[U]Int{16|32|64}|CInt{16|32}|[C]Float{32|64}] [-of <output_format>]
                      [-oo <NAME>=<VALUE>]... [--quiet]
                      <src_datasource> <dst_filename>

... but

C:\v>gdal_rasterize -ts 3 2 -burn 1 -ot Byte D.dpkg D.tiff
ERROR 4: D.dpkg: No such file or directory
rouault commented 2 weeks ago

Shouldn't it be possible to download the binaries that are created for the CI tests?

Cf "GDAL master Conda builds": https://gdal.org/en/latest/download.html#gdal-master-conda-builds

joa-quim commented 2 weeks ago

NOT A COFFEE, MAYBE A SLIP (night here)

It helps when one uses the right file name

C:\v>gdal_rasterize -ts 3 2 -burn 1 -ot Byte D.gpkg D.tiff
0...10...20...30...40...50...60...70...80...90...100 - done.

it's .gpkg, not .dpkg