GDAL processing provider ignores proxy settings

kannes commented 1 year ago

What is the bug or the crash?

When a proxy is set in Options -> Network, QGIS and its tools should use this proxy to ensure connectivity, prevent data leakage, etc. The GDAL processing provider is ignoring these settings though, it's calls to native GDAL/OGR tools happen without the configured proxy settings.

I realize that this is probably pretty complicated and hairy (one would need to explicitly pass proxy options to GDAL), but it caught me by surprise when things did not work in a workflow where at some stage ogr2ogr was called on a WFS source which was inaccessible because of this. I think that users would expect proxy settings to be honored by core providers.

Steps to reproduce the issue

Add a WFS layer to your QGIS project, e.g. ms:inv80_lgeomorph from http://maps.zh.ch/wfs/FnsNSWFS (random result from web search)
Go to Options -> Network
Enable Use Proxy for Web Access
Choose HttpProxy or something else where you can specify a proxy
Set the host to localhost and the port to a port where you do not have a proxy running (e.g. 34567 probably)
(Don't test this right now, Processing would crash when trying to open the tool dialog in the next step (#51958): If you would reload the project now, QGIS should not be able to load the layer, this shows that the proxy setting is correctly applied and the proxy is not working, as intended.)
Execute the GDAL -> Vector information tool (gdal:ogrinfo) for the layer
The tool will run and finish without any issues, providing the remote information of the layer as it bypasses the proxy settings and fetches them from the WFS

The tool should not bypass the proxy settings. For the test case, it should not be able to access a remote source.

Versions

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd">

Supported QGIS version

[X] I'm running a supported QGIS version according to the roadmap.

New profile

[X] I tried with a new QGIS profile

Additional context

Sibling issue for the GRASS provider is #51957, I did not test SAGA or other providers.

jef-n commented 1 year ago

duplicate of #14949

kannes commented 1 year ago

Hm, are you sure? That one seems to have been about layer sources in general, I had no issues there and my test setup did successfully fail to access WFS layers if a broken proxy was set. This one is about calls to the executable binaries, e.g. ogr2ogr, by the processing tools.

edit: Maybe my terminology is wrong, with "provider" I meant "provider of processing tools" :)

pathmapper commented 7 months ago

Just hit this issue, it's not possible to add a COG layer (which uses GDAL) behind proxy:

Test case: https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/Q/WD/2020/7/S2A_36QWD_20200701_0_L2A/TCI.tif

grafik

2024-02-06T11:07:12     INFO    Remote layer : loading /vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/Q/WD/2020/7/S2A_36QWD_20200701_0_L2A/TCI.tif, please wait …
2024-02-06T11:13:03     CRITICAL    Unsupported Data Source : /vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/Q/WD/2020/7/S2A_36QWD_20200701_0_L2A/TCI.tif is not a supported raster data source
             HTTP response code on https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/Q/WD/2020/7/S2A_36QWD_20200701_0_L2A/TCI.tif: 0

This was reported before in https://github.com/qgis/QGIS/issues/42870 (which is closed but not fixed).

I think also related to https://github.com/qgis/QGIS/issues/53351, from @rouault's comment in https://github.com/qgis/QGIS/issues/53351#issuecomment-1593655413:

Those requests are issued by GDAL itself, using curl. To be able to log such requests, it would require changes both in GDAL (similar to the CPLHTTPPushFetchCallback() infrastrcture that was added for the vector side) and QGIS. Not a trivial change

@kannes for the COG issue it's possible to work around with setting the env variable https_proxy. Likely it solves also your reported issue with processing - but I do agree that this should work out-of-the-box.

qgis / QGIS