search

qgis / QGIS

QGIS is a free, open source, cross platform (lin/win/mac) geographical information system (GIS)
https://qgis.org
GNU General Public License v2.0
10.38k stars 2.98k forks source link

Incorrect extent set when importing a NetCDF raster file #49348

Closed NickHumphries closed 2 years ago

NickHumphries commented 2 years ago

What is the bug or the crash?

I have a NetCDF file with a global gridded 3 dimensional data set representing dissolved oxygen concentrations. The dimensions are: Depth, Latitude, Longitude with lengths 75,681,1440. Using an older version of QGIS (e.g. 3.16.0 Hanover) the file imports correctly. I set the CRS to WGS 84 and I can select the depth from a list of bands for the symbology and the map coordinates are correct. In the latest version (3.22.8) , however, the extent information is wrong and the top left pixel of the map is positioned to coordinates 0,0 rather than -180,90. Even if I edit the metadata to manually set the extent it is still wrong. Other simple csv files I import that just have columns of Latitude,Longitude work fine, as does a shape file of the continents. I have tried copying my NetCDF file here, but it won't let me. There is nothing wrong with the particular file I am trying to use, I get the same error with any NetCDF file that has dimensions of Lat & Long. My colleague has the same problem with version 3.22.3.

Steps to reproduce the issue

Import a NetCDF raster file using Layer/Add Layer/Add raster Layer. Select the file and click Add. Then set the CRS.

Versions

This is the version that works:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd">

QGIS version | 3.16.0-Hannover | QGIS code revision | 43b64b13f3 -- | -- | -- | -- Compiled against Qt | 5.11.2 | Running against Qt | 5.11.2 Compiled against GDAL/OGR | 3.1.4 | Running against GDAL/OGR | 3.1.4 Compiled against GEOS | 3.8.1-CAPI-1.13.3 | Running against GEOS | 3.8.1-CAPI-1.13.3 Compiled against SQLite | 3.29.0 | Running against SQLite | 3.29.0 PostgreSQL Client Version | 11.5 | SpatiaLite Version | 4.3.0 QWT Version | 6.1.3 | QScintilla2 Version | 2.10.8 Compiled against PROJ | 6.3.2 | Running against PROJ | Rel. 6.3.2, May 1st, 2020 OS Version | Windows 10 (10.0) Active python plugins | db_manager; MetaSearch; processing QGIS version 3.16.0-Hannover QGIS code revision [43b64b13f3](https://github.com/qgis/QGIS/commit/43b64b13f3) Compiled against Qt 5.11.2 Running against Qt 5.11.2 Compiled against GDAL/OGR 3.1.4 Running against GDAL/OGR 3.1.4 Compiled against GEOS 3.8.1-CAPI-1.13.3 Running against GEOS 3.8.1-CAPI-1.13.3 Compiled against SQLite 3.29.0 Running against SQLite 3.29.0 PostgreSQL Client Version 11.5 SpatiaLite Version 4.3.0 QWT Version 6.1.3 QScintilla2 Version 2.10.8 Compiled against PROJ 6.3.2 Running against PROJ Rel. 6.3.2, May 1st, 2020 OS Version Windows 10 (10.0) Active python plugins db_manager; MetaSearch; processing The version that doesn't is the latest version. ### Supported QGIS version - [X] I'm running a supported QGIS version according to the roadmap. ### New profile - [X] I tried with a new QGIS profile ### Additional context _No response_
saberraz commented 2 years ago

Could you try to add it as a Mesh layer?

NickHumphries commented 2 years ago

Hi Sab,

Mesh layer also goes wrong. With Mesh the map is upside down and still positioned at 0,0.

Cheers,

Nick.

------ Original Message ------ From: "Sab" @.> To: "qgis/QGIS" @.> Cc: "Nick Humphries" @.>; "Author" @.> Sent: Wednesday, 13 Jul, 2022 At 09:11 Subject: Re: [qgis/QGIS] Incorrect extent set when importing a NetCDF raster file (Issue #49348)

Could you try to add it as a Mesh layer? — Reply to this email directly, view it on GitHub https://github.com/qgis/QGIS/issues/49348#issuecomment-1182907337 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AFTJF4XYEVLLY6CV2VVWITDVTZ22LANCNFSM53NY5ZSQ . You are receiving this because you authored the thread.Message ID: @.***>

fengyunyang commented 2 years ago

I have the same problem. And I don't have the mesh layer.

gioman commented 2 years ago

Please attach/link sample data.

NickHumphries commented 2 years ago

Hi Giovanni,

Here is a link to a file that does not map correctly in the latest version of QGIS: https://www.dropbox.com/s/0tbdppbx0p4v83y/20000212.nc?dl=0

Cheers,

Nick.

------ Original Message ------ From: "Giovanni Manghi" @.> To: "qgis/QGIS" @.> Cc: "Nick Humphries" @.>; "Author" @.> Sent: Monday, 8 Aug, 2022 At 18:49 Subject: Re: [qgis/QGIS] Incorrect extent set when importing a NetCDF raster file (Issue #49348)

Please attach/link sample data. — Reply to this email directly, view it on GitHub https://github.com/qgis/QGIS/issues/49348#issuecomment-1208425086 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AFTJF4XZMFNZOORG5JP2Y3LVYFCCFANCNFSM53NY5ZSQ . You are receiving this because you authored the thread.Message ID: @.***>

gioman commented 2 years ago

Using an older version of QGIS (e.g. 3.16.0 Hanover) the file imports correctly. I set the CRS to WGS 84 and I can select the depth from a list of bands for the symbology and the map coordinates are correct. In the latest version (3.22.8) ,

@NickHumphries the older QGIS version you refer probably (likely) used an older version of GDAL and the one that is being used in the latest QGIS releases is reporting that the top left corner is 0,0. You should probably inquiry on the GDAL tracker about this change 


gdalinfo 20000212.nc
Warning 1: dimension #1 (Longitude) is not a Longitude/X dimension.
Warning 1: dimension #0 (Latitude) is not a Latitude/Y dimension.
Driver: netCDF/Network Common Data Format
Files: 20000212.nc
Size is 4320, 2041
Metadata:
  mld#add_offset=-0.1525925546884537
  mld#cell_methods=area: mean
  mld#long_name=Density ocean mixed layer thickness
  mld#scale_factor=0.1525925546884537
  mld#standard_name=ocean_mixed_layer_thickness_defined_by_sigma_theta
  mld#units=m
  mld#unit_long=Meters
  mld#valid_max=19243
  mld#valid_min=1
  mld#_FillValue=-32767
  NC_GLOBAL#bulletin_date=2000-02-16 00:00:00
  NC_GLOBAL#bulletin_type=operational
  NC_GLOBAL#comment=CMEMS product
  NC_GLOBAL#Conventions=CF-1.4
  NC_GLOBAL#Created_By=Nick
  NC_GLOBAL#Created_On=2021-09-17 22:25:04
  NC_GLOBAL#domain_name=GL12
  NC_GLOBAL#easting=longitude
  NC_GLOBAL#field_date=2000-02-12 00:00:00
  NC_GLOBAL#field_julian_date=18304
  NC_GLOBAL#field_type=mean
  NC_GLOBAL#forecast_range=3-day_forecast
  NC_GLOBAL#forecast_type=hindcast
  NC_GLOBAL#history=2017/05/30 17:21:47 MERCATOR OCEAN Netcdf creation
  NC_GLOBAL#institution=MERCATOR OCEAN
  NC_GLOBAL#julian_day_unit=days since 1950-01-01 00:00:00
  NC_GLOBAL#latitude_max=90
  NC_GLOBAL#latitude_min=-80
  NC_GLOBAL#longitude_max=179.91667
  NC_GLOBAL#longitude_min=-180
  NC_GLOBAL#MBA_MLD=V1 2020-03-06
  NC_GLOBAL#northing=latitude
  NC_GLOBAL#references=http://www.mercator-ocean.fr
  NC_GLOBAL#source=MERCATOR GLORYS12V1
  NC_GLOBAL#title=daily mean fields from Global Ocean Physics Analysis and Forecast updated Daily
  NC_GLOBAL#z_max=5727.917
  NC_GLOBAL#z_min=0.49402499
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 2041.0)
Upper Right ( 4320.0,    0.0)
Lower Right ( 4320.0, 2041.0)
Center      ( 2160.0, 1020.5)
Band 1 Block=2160x1021 Type=Int16, ColorInterp=Undefined
  NoData Value=-32767
  Unit Type: m
  Offset: -0.152592554688454,   Scale:0.152592554688454
  Metadata:
    add_offset=-0.1525925546884537
    cell_methods=area: mean
    long_name=Density ocean mixed layer thickness
    NETCDF_VARNAME=mld
    scale_factor=0.1525925546884537
    standard_name=ocean_mixed_layer_thickness_defined_by_sigma_theta
    units=m
    unit_long=Meters
    valid_max=19243
    valid_min=1
    _FillValue=-32767
NickHumphries commented 2 years ago

Hi Giovanni,

Well, it's easier for me to just use the old 3.16 version. No one in our lab can use the latest version, as it doesn't import NetCDF files correctly. Can we get the latest QGIS to use the earlier GDAL (whatever that is) if that is the problem? As no NetCDF rasters can be imported, will this be fixed at some point?

As I've said, I'll just use the old version, as that's fine for now.

Cheers,

Nick.

------ Original Message ------ From: "Giovanni Manghi" @.> To: "qgis/QGIS" @.> Cc: "Nick Humphries" @.>; "Mention" @.> Sent: Tuesday, 9 Aug, 2022 At 10:33 Subject: Re: [qgis/QGIS] Incorrect extent set when importing a NetCDF raster file (Issue #49348)

Using an older version of QGIS (e.g. 3.16.0 Hanover) the file imports correctly. I set the CRS to WGS 84 and I can select the depth from a list of bands for the symbology and the map coordinates are correct. In the latest version (3.22.8) , @NickHumphries https://github.com/NickHumphries the older QGIS version you refer probably (likely) used an older version of GDAL and the one that is being used in the latest QGIS releases is reporting that the top left corner is 0,0. You should probably inquiry on the GDAL tracker about this change gdalinfo 20000212.ncWarning 1: dimension #1 (Longitude) is not a Longitude/X dimension.Warning 1: dimension #0 (Latitude) is not a Latitude/Y dimension.Driver: netCDF/Network Common Data FormatFiles: 20000212.ncSize is 4320, 2041Metadata: mld#add_offset=-0.1525925546884537 mld#cell_methods=area: mean mld#long_name=Density ocean mixed layer thickness mld#scale_factor=0.1525925546884537 mld#standard_name=ocean_mixed_layer_thickness_defined_by_sigma_theta mld#units=m mld#unit_long=Meters mld#valid_max=19243 mld#valid_min=1 mld#_FillValue=-32767 NC_GLOBAL#bulletin_date=2000-02-16 00:00:00 NC_GLOBAL#bulletin_type=operational NC_GLOBAL#comment=CMEMS product NC_GLOBAL#Conventions=CF-1.4 NC_GLOBAL#Created_By=Nick NC_GLOBAL#Created_On=2021-09-17 22:25:04 NC_GLOBAL#domain_name=GL12 NC_GLOBAL#easting=longitude NC_GLOBAL#field_date=2000-02-12 00:00:00 NC_GLOBAL#field_julian_date=18304 NC_GLOBAL#field_type=mean NC_GLOBAL#forecast_range=3-day_forecast NC_GLOBAL#forecast_type=hindcast NC_GLOBAL#history=2017/05/30 17:21:47 MERCATOR OCEAN Netcdf creation NC_GLOBAL#institution=MERCATOR OCEAN NC_GLOBAL#julian_day_unit=days since 1950-01-01 00:00:00 NC_GLOBAL#latitude_max=90 NC_GLOBAL#latitude_min=-80 NC_GLOBAL#longitude_max=179.91667 NC_GLOBAL#longitude_min=-180 NC_GLOBAL#MBA_MLD=V1 2020-03-06 NC_GLOBAL#northing=latitude NC_GLOBAL#references=http://www.mercator-ocean.fr NC_GLOBAL#source=MERCATOR GLORYS12V1 NC_GLOBAL#title=daily mean fields from Global Ocean Physics Analysis and Forecast updated Daily NC_GLOBAL#z_max=5727.917 NC_GLOBAL#z_min=0.49402499Corner Coordinates:Upper Left ( 0.0, 0.0)Lower Left ( 0.0, 2041.0)Upper Right ( 4320.0, 0.0)Lower Right ( 4320.0, 2041.0)Center ( 2160.0, 1020.5)Band 1 Block=2160x1021 Type=Int16, ColorInterp=Undefined NoData Value=-32767 Unit Type: m Offset: -0.152592554688454, Scale:0.152592554688454 Metadata: add_offset=-0.1525925546884537 cell_methods=area: mean long_name=Density ocean mixed layer thickness NETCDF_VARNAME=mld scale_factor=0.1525925546884537 standard_name=ocean_mixed_layer_thickness_defined_by_sigma_theta units=m unit_long=Meters valid_max=19243 valid_min=1 _FillValue=-32767 — Reply to this email directly, view it on GitHub https://github.com/qgis/QGIS/issues/49348#issuecomment-1209144996 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AFTJF4WJOM5JQ7WGI3F7BVTVYIQYPANCNFSM53NY5ZSQ . You are receiving this because you were mentioned.Message ID: @.***>

gioman commented 2 years ago

(whatever that is)

@NickHumphries QGIS is made of many building blocks, onde of them being GDAL/OGR a fundamental library used in QGIS and in many other GIS software. In this case is used to read/write raster data, so if there is a bug in GDAL version that is being used by a specific QGIS release, then also QGIS is affected. And if this is confirmed then the bug must be fixed in the upstream library and not QGIS.

But this must be confirmed anyway, for this reason you should ask in their tracker https://github.com/OSGeo/gdal/issues and ask why in recent releases (QGIS 3.29 uses GDAL 3.5) gdal seems to read incorrectly the coordinate of the UL corner of such datasets.

image

NickHumphries commented 2 years ago

Hi Giovanni,

OK, thanks for the explanation. So, I will log the error with GDAL, but is it possible to backtrack to an earlier, working, version of GDAL in QGIS?

Cheers,

Nick.

------ Original Message ------ From: "Giovanni Manghi" @.> To: "qgis/QGIS" @.> Cc: "Nick Humphries" @.>; "Mention" @.> Sent: Tuesday, 9 Aug, 2022 At 11:34 Subject: Re: [qgis/QGIS] Incorrect extent set when importing a NetCDF raster file (Issue #49348)

(whatever that is) @NickHumphries https://github.com/NickHumphries QGIS is made of many building blocks, onde of them being GDAL/OGR a fundamental library used in QGIS and in many other GIS software. In this case is used to read/write raster data, so if there is a bug in GDAL version that is being used by a specific QGIS release, then also QGIS is affected. And if this is confirmed then the bug must be fixed in the upstream library and not QGIS. But this must be confirmed anyway, for this reason you should ask in their tracker https://github.com/OSGeo/gdal/issues https://github.com/OSGeo/gdal/issues and ask why in recent releases (QGIS 3.29 uses GDAL 3.5) gdal seems to read incorrectly the coordinate of the UL corner of such datasets. image https://user-images.githubusercontent.com/1951107/183627593-e86ac1f6-d29f-465d-a293-9e3c89a0117b.png — Reply to this email directly, view it on GitHub https://github.com/qgis/QGIS/issues/49348#issuecomment-1209207724 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AFTJF4Q27NH3QGCHAI4XMC3VYIX2ZANCNFSM53NY5ZSQ . You are receiving this because you were mentioned.Message ID: @.***>

gioman commented 2 years ago

but is it possible to backtrack to an earlier, working, version of GDAL in QGIS?

@NickHumphries if you can build from source then yes. Anyway I think (not 100% sure) than the latest code versions of QGIS needs a minimum version of GDAL that could also be affected. It would e necessary to check previous GDAL releases, for example by checking your data with the gdalinfo tool https://gdal.org/programs/gdalinfo.html

github-actions[bot] commented 2 years ago

The QGIS project highly values your report and would love to see it addressed. However, this issue has been left in feedback mode for the last 14 days and is being automatically marked as "stale". If you would like to continue with this issue, please provide any missing information or answer any open questions. If you could resolve the issue yourself meanwhile, please leave a note for future readers with the same problem and close the issue. In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this issue. If there is no further activity on this issue, it will be closed in a week.

ar-siddiqui commented 2 years ago

I can confirm a similar issue in 3.24.

Sample NetCDF: https://thredds.daac.ornl.gov/thredds/fileServer/ornldaac/1247/AWT_S_SOC.nc4

image

image

image

QGIS version 3.24.3-Tisler QGIS code revision cf22b74 Qt version 5.15.3
Python version 3.9.5
GDAL/OGR version 3.4.3
PROJ version 9.0.0
EPSG Registry database version v10.054 (2022-02-13)
GEOS version 3.10.2-CAPI-1.16.0
SQLite version 3.38.1
PDAL version 2.3.0
PostgreSQL client version unknown
SpatiaLite version 5.0.1
QWT version 6.1.6
QScintilla2 version 2.13.1
OS version Windows 10 Version 2009
File downloaded and then loaded File loaded through Protocol Windows 10 Version 2009

github-actions[bot] commented 2 years ago

The QGIS project highly values your report and would love to see it addressed. However, this issue has been left in feedback mode for the last 14 days and is being automatically marked as "stale". If you would like to continue with this issue, please provide any missing information or answer any open questions. If you could resolve the issue yourself meanwhile, please leave a note for future readers with the same problem and close the issue. In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this issue. If there is no further activity on this issue, it will be closed in a week.

ar-siddiqui commented 2 years ago

Feedback has been provided.

saberraz commented 2 years ago

Unfortunately these types of issues with netcdf/grib files are common. You need to use cdo tools to correct it:

cdo -invertlat 20000212.nc outfile1.nc
cdo -invertlon  outfile1.nc outfile2.nc

The output should be in the right location: nc

ar-siddiqui commented 2 years ago

@saberraz

https://github.com/qgis/QGIS/issues/49348#issuecomment-1232248931

The same file behaves differently in QGIS if loaded from disk vs HTTP, is that a NetCDF issue too? See my above comment for reference.

saberraz commented 2 years ago

In QGIS nightly, I can see both with no issues in EPSG:4326

alex-s-gardner commented 2 years ago

Hi all, just ran across this thread and I'm having similar issues with QGIS 3.26. NetCDF extents are reported correctly by GDAL 3.4.0, released 2021/11/04, but QGIS 3.26 is unable to read extents correctly.

alex-s-gardner commented 2 years ago

QGIS 3.14 seems to have no issues and loads the NetCDF layer much faster