qgis / QGIS

QGIS is a free, open source, cross platform (lin/win/mac) geographical information system (GIS)
https://qgis.org
GNU General Public License v2.0
10.6k stars 3.01k forks source link

Qgis won't draw raster tiles / hard crash. (mac, linux) #15971

Closed qgib closed 5 years ago

qgib commented 11 years ago

Author Name: Graeme Bell (Graeme Bell) Original Redmine Issue: 6829 Affected QGIS version: master


(Blocker: can't do anything with generated rasters till this is fixed. Unsure if core qgis or postgis-raster bug).

General description: Bugs in raster rendering or extent calcuation. Test case code provided to allow it to be quickly reproduced, and tested on 2 builds of Qgis on two OS (details at end).

Start by disabling 'on the fly reprojection' in qgis.

Bug 1 - no display of most of raster.

  1. run this in psql

drop table bigtest, bigrast; create table bigtest (row_number serial, x int, y int);

insert into bigtest (x, y) select a,b from generate_series(-80000,1120000-1,100000) a, generate_series(6440000,7950000-1,100000) b;

create table bigrast (rid serial primary key, rast raster);

insert into bigrast
select row_number, ST_MakeEmptyRaster(100,100,x,y,1000,-1000,0,0,25833) from bigtest;

update bigrast
set rast=st_addband(rast, '1BB'::text, 1,0);

SELECT AddRasterConstraints('bigrast'::name, 'rast'::name);

This sets up a test raster covering norway in UTM33 at low resolution.

Add the layer to QGis canvas using Database/DB manager.

Expected result: Should be a black box from -80000, 8040000 top left to 1120000, 6440000 bottom right

Actual result:

In both versions of QGIS, generated appearance is a thin black strip from -80000, 6440000 top left to 1120000, 6340000. e.g. one row of tiles, most of the raster is missing.

Layer extent given in properties/metadata as:

-80000, 4840000 112000, 6440000

This is wrong.

In postgis:

select ST_Raster2WorldCoordY(rast,1) from bigrast;

yields correct values.

Notes:

  1. Where is the 4840000 coming from???

  2. I believe qgis may be cropping the raster tiles with an upside down, incorrect extent somehow. Is it calculating extent for itself using pixel scale settings and top left position - rather than via postgis extents? Is it interpreting the 'y-pixel scale' wrong and forgetting to invert it?

I get the same results from the linux 1.8 and Macos 1.9dev versions.

Bug 2

Similar, but this time a positive y-scale pixel size value. This shouldn't work, but does.


drop table bigtest, bigrast;

create table bigtest (row_number serial, x int, y int);

insert into bigtest (x, y) select a,b from generate_series(-80000,1120000-1,100000) a, generate_series(6440000,7950000-1,100000) b;

create table bigrast (rid serial primary key, rast raster);

insert into bigrast
select row_number, ST_MakeEmptyRaster(100,100,x,y,1000,1000,0,0,25833) from bigtest;

update bigrast
set rast=st_addband(rast, '1BB'::text, 1,0);

SELECT AddRasterConstraints('bigrast'::name, 'rast'::name);

LINUX 1.8 Add to qgis with DB Manager -> QGis implodes - hard crash/core dump!

Debug crash output here http://pastebin.com/U3Htd6p7

MACOS 1.9 Displays the correct raster output that was expected for bug1 test case! (but shouldn't)

N.B. using a positive y-scale pixel size is the change. This was an attempt to stop the extents being made 'upside down' as in test1.

However, y-scale should be negative e.g. where it is auto-set by postgis raster:

http://postgis.refractions.net/documentation/manual-2.0/RT_ST_MakeEmptyRaster.html

"The last version use a single parameter to specify the pixel size (pixelsize). scalex is set to this argument and scaley is set to the negative value of this argument. skewx and skewy are set to 0."

also http://en.wikipedia.org/wiki/World_file

"Line 4: E: y component of the pixel height (y-scale), almost always negative"

I believe that a negative y-pixel size is the correct way to represent things. I think there is at least one bug in how qgis interprets y-pixel scale and how it calculates the extents of the coverage.

Tested in Fedora Linux with this build:

QGIS version 1.8.0-Lisboa QGIS code revision exported Compiled against Qt 4.8.2 Running against Qt 4.8.3 Compiled against GDAL/OGR 1.9.1 Running against GDAL/OGR 1.9.1 GEOS Version 3.3.2 PostgreSQL Client Version 9.1.4 SpatiaLite Version 2.4.0 QWT Version 5.2.2

And MacOS X Mountain Lion with this build:

QGIS version 1.9.0-Master QGIS code revision 9dd4a92 Compiled against Qt 4.8.3 Running against Qt 4.8.3 Compiled against GDAL/OGR 1.9.1 Running against GDAL/OGR 1.9.2 GEOS Version 3.3.5 PostgreSQL Client Version 9.1.6 SpatiaLite Version 3.0.1 QWT Version 6.0.1 PROJ.4 Version 480 QScintilla2 Version 2.6.2

$ postgres --version postgres (PostgreSQL) 9.2.1 postgis is version 2.0.1

qgib commented 11 years ago

Author Name: Graeme Bell (Graeme Bell)


1) Tested in windows: qgis 1.9 fcf108d, Mac 1.8.0 Windows 1.8.0 They all behave the same as Mac 1.9dev. So the hard crash only seems to happen in Linux 1.8 version.

But the bad extents/mostly missing tiles exists in all versions/platforms.

2) Upgraded to postgis 2.0.2 (built on gdal 1.10dev) and re-tested. No change.

POSTGIS="2.0.2 r10789" GEOS="3.3.6-CAPI-1.7.6" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10dev, released 2011/12/29" LIBXML="2.8.0" TOPOLOGY RASTER

qgib commented 11 years ago

Author Name: Giovanni Manghi (@gioman)


Graeme Bell wrote:

1) Tested in windows: qgis 1.9 fcf108d, Mac 1.8.0 Windows 1.8.0 They all behave the same as Mac 1.9dev. So the hard crash only seems to happen in Linux 1.8 version.

But the bad extents/mostly missing tiles exists in all versions/platforms.

2) Upgraded to postgis 2.0.2 (built on gdal 1.10dev) and re-tested. No change.

POSTGIS="2.0.2 r10789" GEOS="3.3.6-CAPI-1.7.6" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10dev, released 2011/12/29" LIBXML="2.8.0" TOPOLOGY RASTER

is this a regression in 1.8/master since 1.7.4?


qgib commented 11 years ago

Author Name: Graeme Bell (Graeme Bell)


I'm not sure if this very recent discussion on the Postgis-devel list is relevant here:

http://lists.osgeo.org/pipermail/postgis-devel/2012-November/022914.html

qgib commented 11 years ago

Author Name: Jorge Arevalo (Jorge Arevalo)


Hello,

I'm the GDAL PostGIS Raster driver developer. And I think this is a problem of the driver, actually. The QGIS PostGIS Raster plugin uses it underneath.

First thing I must say: don't use any stable version of GDAL. I committed a version with a lot of bugs fixed in August. It will be included in GDAL 1.10 (due to 2012/12/31). Meanwhile, use the daily snapshot (http://www.gdal.org/daily) or the last version from trunk (https://svn.osgeo.org/gdal/trunk/gdal). Sorry for the inconvenience.

Could you please update your GDAL version to, say, today's snapshot, and test again? The shortest way will be check the raster extent returned by gdalinfo (or the properties of the raster in QGIS)

qgib commented 11 years ago

Author Name: Graeme Bell (Graeme Bell)


Long story short: tried it, no change.

Graemes-MacBook-Pro:~ gbb$ gdalinfo --version GDAL 1.10dev, released 2011/12/29

-> Looks good at the command line...

select postgis_full_version();

POSTGIS="2.0.2 r10789" GEOS="3.3.6-CAPI-1.7.6" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10dev, released 2011/12/29" LIBXML="2.8.0" TOPOLOGY RASTER (1 row)

gdalinfo.... (raster)

Driver: PostGISRaster/PostGIS Raster driver Files: none associated Size is 1200, 1600 Coordinate System is: PROJCS["ETRS89 / UTM zone 33N", GEOGCS["ETRS89", DATUM["European_Terrestrial_Reference_System_1989", SPHEROID["GRS 1980",6378137,298.257222101, AUTHORITY["EPSG","7019"]], TOWGS84[0,0,0,0,0,0,0], AUTHORITY["EPSG","6258"]], PRIMEM["Greenwich",0, AUTHORITY["EPSG","8901"]], UNIT["degree",0.0174532925199433, AUTHORITY["EPSG","9122"]], AUTHORITY["EPSG","4258"]], UNIT["metre",1, AUTHORITY["EPSG","9001"]], PROJECTION["Transverse_Mercator"], PARAMETER["latitude_of_origin",0], PARAMETER["central_meridian",15], PARAMETER["scale_factor",0.9996], PARAMETER["false_easting",500000], PARAMETER["false_northing",0], AUTHORITY["EPSG","25833"], AXIS["Easting",EAST], AXIS["Northing",NORTH]] Origin = (-80000.000000000000000,7940000.000000000000000) Pixel Size = (1000.000000000000000,-1000.000000000000000) Corner Coordinates: Upper Left ( -80000.000, 7940000.000) Lower Left ( -80000.000, 6340000.000) Upper Right ( 1120000.000, 7940000.000) Lower Right ( 1120000.000, 6340000.000) Center ( 520000.000, 7140000.000) Band 1 Block=100x100 Type=Byte, ColorInterp=Gray NoData Value=0 Image Structure Metadata: NBITS=1

BUT

Still does not display correctly in qgis, as before. Also, qgis was loading the raster even when I was using the gdal 1.9.2 without postgis raster support. So... might this be something internal to qgis? Or can I force it to use the gdal library to load the raster?

Interestingly, there is an option in qgis DB manager:

Extent: (unknown) (find out)

(find out) does nothing

QGIS says it is running against '1.9.2' GDAL (I had uninstalled 1.9.2 completely before the rebuild of postgis).

I checked that 1.9.2. was deleted and had rebooted to clear out the shared library from memory.

qgib commented 11 years ago

Author Name: Graeme Bell (Graeme Bell)


Jorge, you are a superhero.

IT WORKS!

The guide at Kyngchaos is ... somewhat ... out of date, as are parts of the configure script used in gdal (--enable-macosx-framework, absent, presumed dead).

Here's what I did to get things working on the mac (was running gdal 1.9, postgis 2.0.1 and qgis 1.9).

--- BACKUP YOUR MAC, your DB, etc: carbon copy cloner or superduper is a good choice ---

rip out the stuff you installed with macports or homebrew port uninstall postgis2 gdal (etc.)

mkdir -p /build/postgis2 cd /build/postgis2 (download, build, install postgis 2.0.2 here e.g. curl http://download.osgeo.org/postgis/source/postgis-2.0.2.tar.gz ./configure; make; make install

to finish upgrading postgis2, run this in psql

CREATE EXTENSION postgis FROM unpackaged; CREATE EXTENSION postgis_topology FROM unpackaged;)

next rebuild gdal2 from svn - do not follow the instructions at kyngchaos.

mkdir -p /build/gdal2 cd /build/gdal2 svn checkout https://svn.osgeo.org/gdal/trunk/gdal ./configure; make; make install

[no fancy configure options / file editing needed... it seems]

next, make sure the old frameworks are REALLY gone (especially if you previously installed the kyngchaos gdal framework packages) Qgis links to these automatically. We want qgis to point to the new library.

mv /Library/Frameworks/GDAL.framework /Library/Frameworks/GDAL-old.framework mkdir -p /Library/Frameworks/GDAL.framework/Versions/1.9/ ln -s /usr/local/lib/libgdal.dylib /Library/Frameworks/GDAL.framework/Versions/1.9/GDAL

Now run qgis, et voila: Bug STOMPED. Tiled rasters render as they should.

Please be warned: this may introduce new horrible bugs elsewhere. But for me, today is a good day.


qgib commented 11 years ago

Author Name: Graeme Bell (Graeme Bell)


n.b. if you have successfully persuaded qgis to run the new version of gdal, it will be reported as 1.10dev in the 'about' menu.

qgib commented 11 years ago

Author Name: Jorge Arevalo (Jorge Arevalo)


Good to hear that! I'll continue working on the driver :-D