Virtual vector does not load correctly on Windows

[pigreco]

I have this Virtual File Format by GDAL/OGR codid19-regioni_dw.vrt

(source : https://data.world/ondata/covid-19-italia-dati-dipartimento-protezione-civile/workspace/file?filename=dpc-covid19-ita-regioni.csv)

it is read well by OGR

ogrinfo codid19-regioni_dw.vrt jgsghstpphjhicstradhy5kpjwrnfy -summary

Layer name: jgsghstpphjhicstradhy5kpjwrnfy
Geometry: Point
Feature Count: 2176
Extent: (0.000000, 0.000000) - (18.171897, 46.499335)
Layer SRS WKT:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0,
        AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.0174532925199433,
        AUTHORITY["EPSG","9122"]],
    AUTHORITY["EPSG","4326"]]
denominazione_regione: String (0.0)
ricoverati_con_sintomi: Integer (0.0)

but not by QGIS!!

step:

1. download the file codid19-regioni_dw.vrt
2. dragAndDrop of the file in QGIS
3. it does not track the dots and the attribute table is empty

---

OSGeo4W64 Win10

Versione di QGIS 3.12.0-București
Revisione codice QGIS cd141490ec
Compilato con Qt 5.11.2
Esecuzione con Qt5.11.2
Compilato con GDAL/OGR 3.0.4
Esecuzione con GDAL/OGR 3.0.4
Compilato con GEOS 3.8.0-CAPI-1.13.1
Esecuzione con GEOS 3.8.0-CAPI-1.13.1 
Compiled against SQLite 3.29.0
Running against SQLite 3.29.0
Versione client PostgreSQL 11.5
Versione SpatiaLite 4.3.0
Versione QWT 6.1.3
Versione QScintilla2 2.10.8
Compiled against PROJ 6.3.1
Running against PROJ Rel. 6.3.1, February 10th, 2020
OS Version Windows 10 (10.0)

[saberraz]

Works fine in master.

[pigreco]

@saberraz

even in the master it doesn't work!

[saberraz]

How are you adding the file? As this:

(the attribute table in this method is not correct for me, but the geometry appears correct): Or download the vrt and add it from browser panel as a file?

[pigreco]

@saberraz

thanks for your tests, but I can't even use a new profile.

thanks anyway :-(

[pigreco]

@saberraz

maybe it works on linux but not on win ??

[stev-0]

Doesn't work on Windows for me.

[albertograva]

On ubuntu 18.04 and Qgis 3.10 it works!

[gioman]

On ubuntu 18.04 and Qgis 3.10 it works!

QGIS 3.10 on Ubuntu 18.04 still uses GDAL 2.4 while on Windows is on GDAL 3, this maybe the cause of the difference you are seeing.

[saberraz]

I tried on QGIS 2.18 with gdal 2.x in windows and it also failed.

I wonder if this is something to do with the way OS handles HTTPS certificate.

[pigreco]

I think it's a bug, please remove Feedback tag, thanks

[gioman]

I think it's a bug, please remove Feedback tag, thanks

@pigreco we are not done testing, be patient.

[gioman]

Translating the vrt with ogr2ogr works as expected, so all the infos seems to point at some bug in QGIS on Windows OR something Windows related. I can't test on macOS at the moment (I will as soon as possible).

[ivn951]

@pigreco on macOS it work...(QGIS 3.12)

[saberraz]

Translating the vrt with ogr2ogr works as expected, so all the infos seems to point at some bug in QGIS on Windows OR something Windows related. I can't test on macOS at the moment (I will as soon as possible).

The problem is, the source gets updated and if you translate it, you need to run a cron job to keep the translated version in sync with the source :)

[gioman]

The problem is, the source gets updated and if you translate it, you need to run a cron job to keep the translated version in sync with the source :)

yes I know, but that was not my point. What I wanted to test was if OGR had problems reading the data.

[marcobra]

Works on: Ubuntu 18.04.4 LTS Release: 18.04 Codename: bionic Versione di QGIS : 3.12.0-București Compilato con Qt: 5.9.5 | Esecuzione con Qt : 5.9.5 Compilato con GDAL/OGR: 2.2.3 | Esecuzione con GDAL/OGR : 2.2.3

[aborruso]

Also for me it does not work: I have QGIS 3.10.2-A Coruña and GDAL 3.0.3, in Windows.

Running the ogr command from OSGEO shell I have no problem

[pigreco]

Please, if you need to make a fundraiser say it explicitly. thanks

[gioman]

Please, if you need to make a fundraiser say it explicitly. thanks

@pigreco there is no "you" here, just "we". If there is a bugs that annoys you particularly and is not getting attention/traction from the project than you should consider fixing it one way or another. For example the QGIS Portugal user group organized a small crowd funding a few weeks ago and supported fixing an annoying bug (for us, at least): https://github.com/qgis/QGIS/issues/35465

[nyalldawson]

@gioman I think @pigreco was offering to fund a fix?

[elpaso]

I know it doesn't help but it works fine for me on current master/linux built with GDAL 3.

[pigreco]

I just did a test with the same file attached on the master OSGeo4W64 win 10 QGIS Does not work!

[gioman]

I know it doesn't help but it works fine for me on current master/linux built with GDAL 3.

yeah it seems to be a Windows only issue...

[elpaso]

I can have a look, if just debugging and building on :windo wasn't so time consuming... :unamused:

[elpaso]

And, surprise! I cannot reproduce on windows/current master:

[agiudiceandrea]

Different GDAL / PROJ versions?

[elpaso]

Different GDAL / PROJ versions?

Yeah, possibly.

[elpaso]

Ok, I've spent the last 8 hours rebuilding QGIS on windows with latest GDAL and PROJ, well, my CPU did that, I feel guilty for all the wasted energy because it works :confused: :

[pigreco]

@elpaso

Thanks for the various tests, but in my machine it doesn't work even on clean QGIS profile :-(

[pigreco]

Hi @elpaso you have compiled and therefore I assume it is a different thing from using my same version. Please, if you can, use the same version as me. thank you

[elpaso]

Hi @elpaso you have compiled and therefore I assume it is a different thing from using my same version. Please, if you can, use the same version as me. thank you

I could but I don't see the point: if I can reproduce it within a debugger I can try to fix it but by running a release build I cannot do nothing anyway.

[gioman]

rebuilding QGIS on windows

@elpaso where I can find clear instructions on how to compile on Windows?

[pigreco]

@elpaso

I could but I don't see the point: if I can reproduce it within a debugger I can try to fix it but by running a release build I cannot do nothing anyway.

If useful, I can make my laptop available for remote testing

[agiudiceandrea]

I did some tests with the VRT layer file codid19-regioni_dw.vrt at https://github.com/pigreco/COVID-19_ITA/blob/master/risorse/codid19-regioni_dw.vrt using the following QGIS versions:

• on Windows 10: QGIS 3.14.0 and 3.10.6, both with GDAL 3.0.4 / PROJ 6.3.2, and QGIS 3.4.15 (GDAL 2.4.3 / PROJ 5.2.0) all fail to display the layer
• on Windows 7: QGIS 3.10.1 (GDAL 3.0.2 / PROJ 6.2.1) and 3.4.5 (GDAL 2.4.0 / PROJ 5.2.0) both fail to display the layer
• on Ubuntu Linux 20.04 (VirtualBox hosted by Windows 10): QGIS 3.14.0 (GDAL 3.0.4 / PROJ 6.3.1) succeeds to display the layer
• on Ubuntu Linux 18.04 (VirtualBox hsted by Windows 10): QGIS 3.14.1 (GDAL 2.2.3 / PROJ 4.9.3) succeeds to display the layer

[pigreco]

Excuse my reflection, but the problem is not to be found in GDAL or PROJ but in windows, as shown by the various tests.

regards

[agiudiceandrea]

Yes, the different GDAL/PROJ (mostly GDAL) version or a local issue were initial hypotheses. Testing various combination of QGIS/GDAL version on both Windows and Linux on different systems was a way of trying to rule out those hypotheses: in fact, it seems now very unlikely, since all the combination of QGIS/GDAL on Windows fail and all those on Linux succeed.

[elpaso]

rebuilding QGIS on windows

@elpaso where I can find clear instructions on how to compile on Windows?

I don't know if they qualify for clear and YMMV but here they are https://github.com/qgis/QGIS/blob/master/INSTALL.md#4-building-on-windows

Note that it took me several years and an endless combination of black magic, sorcery, blood & tears, payers & blasphemies to have a kind-of-working windows toolchain. As a friend, I strongly advise you to go that route and I'm wondering what will you gain from it anyway, unless you plain to do some debugging (that does not even work well).

[elpaso]

@elpaso

I could but I don't see the point: if I can reproduce it within a debugger I can try to fix it but by running a release build I cannot do nothing anyway.

If useful, I can make my laptop available for remote testing

@pigreco just to be clear, I absolutely trust you and all the other users that are experiencing the issue and that the issue does exist, I'm sorry I cannot do anything about it because I cannot even reproduce it on my machine, and that is the prerequisite for me to even try to debug it.

[elpaso]

@pigreco out of curiosity, did you try if the MXE builds are also affected from this issue? They use slightly different versions of the libraries.

[gioman]

Note that it took me several years and an endless combination of black magic, sorcery, blood & tears, payers & blasphemies to have a kind-of-working windows toolchain. As a friend, I strongly advise you to go that route and I'm wondering what will you gain from it anyway, unless you plain to do some debugging (that does not even work well).

@elpaso I read this 5 minutes ago and still laughing! :)

[agiudiceandrea]

Hi @elpaso, great suggestion! Latest qgis-dev 3.15.0-64 (8d2a0d1ebb) fails to correctly load the VRT layer,

while the latest MXE at the same commit (8d2a0d1ebb) succeeds!

[elpaso]

@agiudiceandrea unfortunately that doesn't help much, MXE is using old GDAL and PROJ.

[agiudiceandrea]

@elapso, If it fails on Windows with an OSGeo4W QGIS version, while it works with an MXE QGIS version and with Ubuntu and Mac versions, the issue could be related to OSGeo4W.

In fact, I found the culprit!

In OSGeo4W the environmental variable VSI_CACHE (GDAL related) is set to TRUE. In the MXE build this env var is no set (so the default FALSE is assumed by GDAL). Setting VSI_CACHE to TRUE before launching on the MXE QGIS build lead to the same issue and the VRT layer is not displayed. Removing the VSI_CACHE setting (or setting it to FALSE) from the OSGeo4W launching procedure, the issue is not present and the VRT layer is correctly displayed.

[agiudiceandrea]

@pigreco, @stev-0, could you please check if the following procedure fixes the issue for you?

In QGIS, go to Settings->Options->System, in the "Environment" box check "Use custom variables", then add a new row with Overwrite in the "Apply" column, VSI_CACHE in the "Variable" column and FALSE in the "Value" column.

Restart QGIS and try to add the VRT layer.

[pigreco]

@agiudiceandrea @elpaso bingo

[elpaso]

@agiudiceandrea congratulations! We have a very good team of investigators here :smiley:

[agiudiceandrea]

The investigation is not over... :-)

It seems the VSI_CACHE=TRUE custom setting in OSGeo4W (along with VSI_CACHE_SIZE=1000000) was introduced 7 years ago https://github.com/qgis/QGIS/commit/9222f152b30948a3a9d9f66dffd162548022f7fd by @blazek to workaround a reported performance issue about shapefile reading over network on Windows. See https://github.com/qgis/QGIS/issues/15688 (https://issues.qgis.org/issues/6448) and this thread on gdal-dev ml.

Is it still required, since GDAL/OGR may have been improved in the meantime?

Anyway, looking at the HTTPS traffic sniffed during the QGIS attempt to retrieve the data from the server, it seems also that the GDAL/OGR VSI cache functionality does not work well with the way the data is provided by the server at download.data.world, while it works well for data stored on other servers (e.g. githubusercontent.com) even with VSI_CACHE set to TRUE.

I will try to ask to the gdal-dev ml for clarification.

[agiudiceandrea]

The handle of data hosted on data.world server if VSI_CACHE=TRUE was fixed upstream in GDAL/OGR. See https://github.com/OSGeo/gdal/pull/3007.

So I think this issue report could be closed now.

[pigreco]

@agiudiceandrea @elpaso grazie mille!