Closed mattenvsys closed 4 years ago
This isn't a qgis issue -- if gdal is reporting it as a boundcrs, then qgis will do the same. You'll need to file this upstream with gdal (and verify if indeed the layer IS in the correct crs -- chances are it's not!)
Also on 3.12 and master. Is fine on 3.4.15.
@nyalldawson Hard to tell if it is a bug and where it belongs to :-)
If you look at the inside of the TIFF file with low level listgeo
utilitiy, you can see that TOWGS84 clause is encoded in it, so techincally when reading back this file, a BoundCRS is expected. This was the historical behaviour of GDAL before 3.0.3, when writing a GeoTIFF file, to write the TOWGS84 parameter that were associated with a EPSG code when read by GDAL importFromEPSG(), and has been changed since. But when you read back such file, you don't know if the TOWGS84 is intended (the user really wants those parameters to be used), or just because it was written by default. So this is mostly an historical artifact due to how CRS transformation worked in the past pre PROJ6/GDAL3 and how GDAL dealt with that.
$ listgeo 27700.tif
[ ... snip ... ]
Keyed_Information:
GTModelTypeGeoKey (Short,1): ModelTypeProjected
GTRasterTypeGeoKey (Short,1): RasterPixelIsArea
GTCitationGeoKey (Ascii,34): "OSGB 1936 / British National Grid"
GeogCitationGeoKey (Ascii,10): "OSGB 1936"
GeogAngularUnitsGeoKey (Short,1): Angular_Degree
GeogTOWGS84GeoKey (Double,7): 446.448 -125.157 542.06
0.15 0.247 0.842
-20.489
ProjectedCSTypeGeoKey (Short,1): PCS_British_National_Grid
ProjLinearUnitsGeoKey (Short,1): Linear_Meter
End_Of_Keys.
End_Of_Geotiff.
PCS = 27700 (OSGB 1936 / British National Grid)
[ ... snip ... ]
TOWGS84: 446.448,-125.157,542.06,0.15,0.247,0.842,-20.489
[ ... snip ... ]
Thinking out loud: Maybe QGIS could have some option where the user could decide to use the base CRS of the BoundCRS ?
Hi @rouault , is it possible for this case to be reopened to allow for more visibility?
I still maintain there's no issue in qgis. Users can already manually change a layers crs if they need, but for the automatic crs detection we rely entirely on what gdal gives. And if @rouault says that gdal is giving the right result for this particular layer, then there's nothing else we can do in this situation.
Hi @rouault. Thank you very much for investigating this problem. I work with the original reporter and am investigating the problem to hopefully find a solution.
I have used the listgeo
tool to investigate the file. I have also re-assigned the projection to EPSG:27700 with GDAL 3.0.4 to assess the difference that you suggest.
I notice that the GeogTOWGS84GeoKey is indeed missing from the GDAL 3 version of the file, which was present in the original, and the details seem to now be reported in the ABRIDGEDTRANSFORMATION section of the projection defintion.
Could you please suggest a way of going forward? I understand if this is the new intended behaviour of GDAL, but it's clear that old definitions are not being recognised as a valid EPSG code that they once were. It would be quite a task to re-assign the new projection definition on all previously created files.
And if @rouault says that gdal is giving the right result for this particular layer
The thing is that I'm not sure which is the "right" result. Depending on use cases, reporting the BoundCRS or just the base CRS of it is the right behavior. We cannot know if the TOWGS84 inserted in the geotiff definition was intended or automatic. A possible behaviour would be for GDAL to have a list of the default TOWGS84 parameters per EPSG code that were used in GDAL 2, and if a GeoTIFF file is read with such TOWGS84 parameters, ignore them on reading assuming that the probability they were inserted automatically is high. But there would potentially be workflows when, even when that was done automatically, people would have relied in the past on that. For example, if that file was created from reprojection from another datum, then those TOWGS84 parameters have been used by gdalwarp, and thus when reprojecting it to something else, you'd better use them than the OSTN15 grid.
@rouault ,this indirectly connects back to this idea of adding a GDAL2 authority with a list of towgs84 transformation from that era, and make use of it.
I have run some more tests building Flatpak edition of QGIS on Ubuntu with different versions of GDAL and QGIS.
The tests were built using GDAL 2.4.4, GDAL 3.0.4, QGIS 3.4.15, QGIS 3.10.3
The following combinations all read the EPSG correctly (EPSG:27700) QGIS 3.4.15 with GDAL 2.4.4 QGIS 3.4.15 with GDAL 3.0.4 QGIS 3.10.3 with GDAL 2.4.4
The following combination is incorrect and note the projection as "Unknown CRS" QGIS 3.10.3 with GDAL 3.0.4
This is the same/similar combination that the Windows builds use.
Perhaps there has been some recent regression with how QGIS interacts with GDAL between 3.4 and 3.10?
I've just written the https://github.com/OSGeo/gdal/blob/master/gdal/swig/python/samples/gdal_remove_towgs84.py script. You can download it and run it with GDAL 2 or GDAL 3 to remove, in-place, TOWGS84[] clauses from GeoTIFF file whose CRS contains both a EPSG code and a TOWGS84[] clause. Fixing such files is the way forward.
Hi @rouault. Thank you so much for taking the time in investigating this and providing code; I really appreciate it.
Further to my comment above, I have also tested QGIS 3.10.1 and GDAL 3.0.4. I have found that the issue of "Unknown CRS" is not present in this build.
I am aware that this QGIS blog post highlights that some major changes were made in QGIS 3.10.2 regarding GDAL and PROJ. It is possible that this particular issue was introduced there?
It is possible that this particular issue was introduced there?
There has been some changes at some point in how QGIS reports the SRS it gets from GDAL. But I don't remember the details. In anycase, there's no right & wrong behaviour per se. This is a matter of making a choice that can be the right one in some context, and less appropriate in others. So if in your use case, QGIS behaviour is not the one you expect, better fix your files to remove the offending TOWGS84.
Hi @rouault . It's not just our data which is projecting incorrectly. Here is a zip file of a vector dataset generated and maintained by the UK Government. The source can be found here.
This dataset projects correctly in QGIS 3.4.15 but is roughly 2-5 metres out in QGIS 3.10.3. This is an issue when you expect the data to align with features from another dataset:
QGIS 3.10.3
QGIS 3.4.15
We appreciate the intended fix, but this would involve applying it to all files that have ever been created with GDAL 2 and potentially from other sources, which seems fairly unachievable. EPSG:27700 is a fairly standard projection for the United Kingdom so it's likely that this issue will be affecting many others too.
While I understand that we must go forward I also think that this should really be reopened: QGIS bug or not we are going to take most of the blame for this problem.
@gioman there's nothing we can do on qgis side. We don't mess with what gdal or proj give us, and rely entirely on what they report.
Hi @nyalldawson, can I point you to this post which states that QGIS 3.10.0 and 3.10.1 built with the latest versions of PROJ and GDAL place the data in the correct place and read the projection correctly.
The issue raises itself in QGIS 3.10.2 and beyond.
This blog post states the upgrade "led to a number of regressions" which are "addressed in 3.10.2." Has attempting to fix the initial regressions in 3.10.2 led to different issues presenting itself, such as this one?
@mattenvsys the behaviour pre 3.10.2 was a bug, because a bound crs is not equivalent to the crs it's bound to. In 3.10.1 we incorrectly made this assumption. This was fixed in 3.10.2 with the side effect of this issue (note: not a bug - gdal reports a boundcrs, so we rightly use the same).
Don't have much to add other than that this also affects our users quite considerably too since we generate a lot of GeoTIFFs (global flood mapping) using GDAL 1.x/2.x and it isn't really feasible to apply the suggested fix to all previously generated data (let alone external data, such as that highlighted by @mattenvsys).
I understand that this is a difficult situation, given that QGIS just uses what GDAL provides it with and that it isn't a bug in GDAL. However, it does seem to be a regression in terms of usability. Is there any way around this at all within QGIS itself? For example, (and as much as it pains me to suggest this) something like special casing it, if that's even possible?
An optional behaviour in GDAL or QGIS to extract the base CRS of a bound CRS is something that can be conceived. What would remain to decide is:
But I'd note that the python script I provided edits files in place, altering just the metadata part. It should be able to fix several files per second on a single machine.
What I'm not sure to understand too is that with older QGIS and GDAL, I'd have expected the TOWGS84 values to be also used when reprojection is needed, so I'm not sure why QGIS 3.10 + GDAL 3 would make a difference here in behaviour. Obviously I miss something
- in GDAL and/or QGIS ? and "where" in the software. For GDAL, could be potentially limited to the GTiff driver and perhaps if the TOWGS84 values are known to be the default ones used by GDAL 1/2
I think ideally it would go in GDAL, since then tools like gdalinfo etc would report the "expected" base CRS rather than the "unexpected" bound CRS. Unfortunately, I don't think it can be limited to just the GeoTIFF driver, as this appears to be a wider issue (see f.ex. this comment from @mattenvsys where it is affecting Shapefiles too).
- which default behaviour to select, that is return the bound CRS or the base CRS of it.
For us, returning the base CRS would definitely be preferred. I would imagine that this would also be the majority view too, given that this was the default behaviour previously (even if it was technically wrong).
- and find people to implement that
I could ask if this would be something that our company would be willing to sponsor if that would help? QGIS obviously accepts sponsors and donations, but I don't know how that would work with respect to GDAL?
Would it be better if we continued this discussion on the GDAL issue?
I could ask if this would be something that our company would be willing to sponsor if that would help? QGIS obviously accepts sponsors and donations
@peetw QGIS does not accept donations for targeted bug fixing, but for that you can hire a developer or a company that offers commercial support.
@nyalldawson This may be a silly question, but are there any cases where adding a layer that had a bound CRS to QGIS would result in QGIS correctly recognising the CRS (rather than just saying "Unknown CRS")?
If a bound CRS is sometimes a valid CRS when adding a layer to QGIS, perhaps QGIS could only fall back to the base CRS if the bound CRS is not recognised? I don't know how difficult this would be to implement, but it might just be good enough?
This may be a silly question, but are there any cases where adding a layer that had a bound CRS to QGIS would result in QGIS correctly recognising the CRS (rather than just saying "Unknown CRS")?
@rouault would probably know better than me here, but my gut feeling is no. Explicitly bound CRSes are custom CRSes, they don't match official definitions from the registries.
If a bound CRS is sometimes a valid CRS when adding a layer to QGIS, perhaps QGIS could only fall back to the base CRS if the bound CRS is not recognised? I don't know how difficult this would be to implement, but it might just be good enough?
It's possible, but I'm totally against this approach. It would mean that you'll get different results in QGIS vs other open source tools (such as GDAL, GRASS, etc).
QGIS was historically filled with all kinds of fragile stick-tape hacks to handle various CRS identification issues. When we ported QGIS to PROJ 6 it was done with the principal "NO MORE DOWNSTREAM HACKS, EVER!", and I'm extremely reluctant to backtrack on this. We should instead implicitly trust what the dedicated libraries for datasource and projection handling give us (i.e. gdal and proj) and ensure that any edge cases are handled in these libraries and not in QGIS alone.
I hate to be a road block here, but this really needs to be handled in gdal and not QGIS.
@rouault would probably know better than me here, but my gut feeling is no. Explicitly bound CRSes are custom CRSes, they don't match official definitions from the registries.
While in theory, the WKT2:2019 standard allows an identifier to be attached to a BoundCRS itself, the EPSG database doesn't have BoundCRS. So in practice you will never get a BoundCRS that has an identifier. Only its base CRS may have one.
I hate to be a road block here, but this really needs to be handled in gdal and not QGIS.
I agree with your general philosophy. I'm not sure however if that can be handled in GDAL in a fully automated way. There are use cases where reporting only the base CRS can make sense, and other where reporting the boundCRS is the right thing to do. Only the end user can know... GDAL could decide to report only the baseCRS, but I'm pretty sure other people would complain that it breaks their use cases.
But here, I don't understand why the situation is worse with latest QGIS than it was before. Before it used the TOWGS84 transform of the GeoTIFF file when reprojecting, and now it also does, right ? I miss something
But here, I don't understand why the situation is worse with latest QGIS than it was before. Before it used the TOWGS84 transform of the GeoTIFF file when reprojecting, and now it also does, right ? I miss something
I suspect the difference is that since the CRS doesn't get detected as EPSG:27700, the available coordinate operations for the transform don't include those that utilise a grid shift
There are use cases where reporting only the base CRS can make sense, and other where reporting the boundCRS is the right thing to do. Only the end user can know...
@rouault
There are use cases where reporting only the base CRS can make sense, and other where reporting the boundCRS is the right thing to do. Only the end user can know...
In that case possibly we should just end this discussion. You've made an automated tool available to repair the files, and QGIS users have the ability to manually change the autodetected CRS if needed. There's nothing more we can do from QGIS, and if there's nothing more we can do from gdal either, then there's just no solution possible...
Hi @nyalldawson, The thing is QGIS doesn't currently inform the user that it is reading the dataset as Unknown CRS. If the project is already in 27700 then the Unknown CRS is added afterwards, it is positioned incorrectly compared to the rest of the data.
If the user then performs processing tasks on these layers, they are then permanently locked in this incorrect position and causes overlaps and slivers to form in data which wouldn't otherwise be there
It's possible, but I'm totally against this approach. It would mean that you'll get different results in QGIS vs other open source tools (such as GDAL, GRASS, etc).
QGIS was historically filled with all kinds of fragile stick-tape hacks to handle various CRS identification issues. When we ported QGIS to PROJ 6 it was done with the principal "NO MORE DOWNSTREAM HACKS, EVER!", and I'm extremely reluctant to backtrack on this. We should instead implicitly trust what the dedicated libraries for datasource and projection handling give us (i.e. gdal and proj) and ensure that any edge cases are handled in these libraries and not in QGIS alone.
I hate to be a road block here, but this really needs to be handled in gdal and not QGIS.
@nyalldawson Honestly, as a fellow developer with a passion for clean, maintainable code, I fully understand your reluctance here and as a contributor to open-source projects myself, I don't want to sound like we're "demanding" anything. However, there does seem to be a clear regression in usability between the last two LTR (3.4.15 and 3.10.4) releases of QGIS (with the understanding that the issue may be caused by/be better fixed in GDAL, not QGIS).
To demonstrate the issue as clearly as possible, I generated sample GeoTIFF and Shapefile (polygon of the GeoTIFF extent) files using QGIS 3.4.15 (GDAL 2.4.3) and QGIS 3.10.4 (GDAL 3.0.4). The data is available here: Data.zip. I then performed the following steps in both QGIS 3.4.15 and QGIS 3.10.4:
For QGIS 3.4.15, the CRS was correctly identified for all layers and all layers aligned correctly:
However, for QGIS 3.10.4, the CRS for gdal_2.tif was "Unknown" (for the reasons discussed previously in this thread) and as a result it was misaligned with respect to the other layers, regardless of which transformation was selected when adding the layer to the project:
The only way to get the gdal_2.tif layer to align correctly was to either manually set its CRS to the project CRS (EPSG:27700) or to remove the TOWGS84 params from the file itself using the script provided.
However, as we've already discussed, the script to remove the TOWGS84 params is not a viable general solution. It may be fine in small or single-man shops, where all generated and incoming data can be carefully controlled, but for large organisations it is just not feasible, given the amount of archived, incoming and constantly generated data (you'd essentially be playing whack-a-mole). It would also require all users to be made aware of the issue and workaround, which is just not possible to enforce.
Manually setting the layer's CRS is also not a viable solution, since once a user has added an "Unknown" (i.e. GDAL < 3) layer to a project and selected a transformation, there is no warning or indication at all when another "Unknown" layer (of the same projection) is then added to the project (since it just silently uses the previously selected transformation from the first "Unknown" layer) so it is not clear to users that they need to go and manually set the CRS for that newly added layer (as also illustrated by @mattenvsys comment). You can test this by creating a copy of gdal_2.tif and adding it as a "step 7" when reproducing the issue in QGIS 3.10.4.
So, where does that leave us? Hopefully we can agree that something needs to be done, whether that is in QGIS or GDAL. Now, as for the what, I am not sure.
@rouault The suggestions to modify GDAL to return the base CRS by default when default TOWGS84 params (i.e. those automatically added by GDAL 1/2) are present (with perhaps a command line switch to return the full bound CRS if required) would seem to me to be perhaps the best compromise, given that it would solve the issue for the majority of users, whilst still allowing the more niche (less common?) use cases.
I've started a discussion thread regarding this on GDAL mailing list: https://lists.osgeo.org/pipermail/gdal-dev/2020-March/051881.html
@rouault Great, thank you - really appreciate it! The linked issue over on GDAL re: Indian transformations is also interesting.
Not sure if it's relevant, but EPSG:27700 happens to be in the list of 370 CRS's discussed here.
Not sure if it's relevant, but EPSG:27700 happens to be in the list of 370 CRS's discussed here.
I just took four CRS's at random from the results.csv. Made some raster data in 3.4, then opened them in 3.10. All four were read as Unknown CRS.
Has the time come to make a post on the official blog stating if you are using any of the listed CRS's in the spreadsheet in 3.10 then you need to be extra vigilant ensuring your data is correctly positioned? Users need to be made aware of this and the current fix to correct it.
GDAL pull request addressing this topic: https://github.com/OSGeo/gdal/pull/2370
Confirming that OSGeo/gdal#2370 solves this issue
@nyalldawson That's excellent news, thank you for testing! And of course many thanks to @rouault for implementing the fix!
@nyalldawson great stuff. Does this work with the current release of GDAL and therefore the current point release of QGIS?
I have tested GDAL 3.1rc2 and can confirm that both new and old specifications of ESPG:27700 are read as expected. Thank you very much @rouault for implementing a solution and @nyalldawson for testing. I really appreciate your efforts.
Hi,
I've got same issue with QGIS 3.12.2 GDAL 3.0.4 and Proj 7.0 Data in 3035 are not well recognize it depends if it's label ESPG 3035 from ETRS89 / LAEA Europe to ETRS89-extended / LAEA Europe. If I load on a recent QGIS 3.12 a layer make last year, it will display unknown SRC and also if I generate with QGIS 3.12 a layer in 3035 as ETRS89-extended an old LTR (3.4 for example) won't recognize it.
Best,
Hi @Doctor-Who. It's possible that you're issue will have also been fixed in GDAL 3.1, which was released a few days ago.
I'm unsure when this will appear in future QGIS releases, but hopefully will do soon.
Data in 3035 are not well recognize it depends if it's label ESPG 3035 from ETRS89 / LAEA Europe to ETRS89-extended / LAEA Europe.
EPSG has made changes in naming of that, but I see there's an alias for the old name. Could you attach a dataset that reproduces that issue ? If it is a shape with a .prj, you can directly file an issue at https://github.com/OSGeo/PROJ/issues with the .prj content (try projinfo --identify @the.prj
)
Hi @anthony-scarth As we using on openSUSE, if we update GDAL package to the 3.1, then all QGIS version will be rebuild and fix :)
Hi @rouault the name as changed but not EPSG but it seems QGIS read alias and not epsg because it said src unknown I've check issue and it's mainly about GPKG and also about Raster data. I can upload a GPKG but not imagery one.
I can upload a GPKG but not imagery one.
That's fine, but perhaps create another dedicated issue as this one is becoming confusing
Where is the right place, I'm quite confusing ? GDAL or PROJ issue ?
Where is the right place
I don't know yet :-) Might be a QGIS one as well. Just open a QGIS issue for now
@rouault I've really try to understand, but i feel mad .. As you suggest, I open a new issue #36699
Raster and some vector files known to be projected in EPSG:27700 are read as "Unknown CRS" in QGIS 3.10.3 running against PROJ 6 and GDAL 3
This affects all raster files created using QGIS 3.4.X / GDAL 2.4.3 / PROJ 5.2 and previous versions, as far as we have tested.
The following projection info is reported by
gdalinfo
on the same file (attached here 27700.zip):GDAL 2.4.3 / PROJ 5.2.0
GDAL 3.0.4 / PROJ 6.3.1
The latter does not interpret as EPSG:27700 in QGIS 3.10.3, but rather as "Unknown CRS". Its position is offset from other files that have projections read correctly.
How To Reproduce
1) Download test file 27700.zip 2) Load into a new instance of QGIS 3.10.3 on Windows 3) The project and file projection should be labelled as "Unknown CRS"
QGIS and OS versions
OS Info: Windows 10 Pro Version 1903 OS Build 18362.693
QGIS version 3.10.3-A Coruña QGIS code revision 0e1f846438 Compiled against Qt 5.11.2 Running against Qt 5.11.2 Compiled against GDAL/OGR 3.0.4 Running against GDAL/OGR 3.0.4 Compiled against GEOS 3.8.0-CAPI-1.13.1 Running against GEOS 3.8.0-CAPI-1.13.1 Compiled against SQLite 3.29.0 Running against SQLite 3.29.0 PostgreSQL Client Version 11.5 SpatiaLite Version 4.3.0 QWT Version 6.1.3 QScintilla2 Version 2.10.8 Compiled against PROJ 6.3.1 Running against PROJ Rel. 6.3.1, February 10th, 2020 OS Version Windows 10 (10.0) Active python plugins envsysCoastal3; envsysSend2google_earth; GroupStats; mapswipetool_plugin; mmqgis; OSTranslatorII; pointsamplingtool; qdraw; qfieldsync; Qgis2threejs; SemiAutomaticClassificationPlugin; splitmultipart; db_manager; MetaSearch; processing