ArcGIS Feature Service data source: can't connect to service that needs custom HTTP referer

[MatzFan]

Description

I am unable to connect to an ArcGIS Feature Server that is hosted (proxied) on the Azure web application platform.

To reproduce

• Right click on the ArcGIS Feature Service tool in the Browser window, give the connection a name and add the following URL
• https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Historic_Buildings/Historic_Buildings/FeatureServer
• Add the following as the HTTP referer https://www.gov.je//citizen/Planning/Pages/HistoricEnvironmentDetail.aspx
• Click "OK"
• In the drop down, see the resulting error "Connection failed: Network error"
• Hover over this to see the message "Error transferring https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Historic_Buildings/Historic_Buildings/FeatureServer&f=json - server replied: Bad Request"

System

QGIS version	3.16.0-Hannover	QGIS code revision	43b64b13f3
Compiled against Qt	5.12.8	Running against Qt	5.12.8
Compiled against GDAL/OGR	3.0.4	Running against GDAL/OGR	3.0.4
Compiled against GEOS	3.8.0-CAPI-1.13.1	Running against GEOS	3.8.0-CAPI-1.13.1
Compiled against SQLite	3.31.1	Running against SQLite	3.31.1
PostgreSQL Client Version	12.4 (Ubuntu 12.4-0ubuntu0.20.04.1)	SpatiaLite Version	4.3.0a
QWT Version	6.1.4	QScintilla2 Version	2.11.2
Compiled against PROJ	6.3.1	Running against PROJ	Rel. 6.3.1, February 10th, 2020
OS Version	Ubuntu 20.04.1 LTS
Active python plugins	ViewshedAnalysis; MetaSearch; processing; db_manage

Notes

I posted this as a question here: https://gis.stackexchange.com/questions/379346/qgis-connect-to-arcgis-featureserver-hosted-behind-azure-proxy

As far as I can tell QGIS is interpreting the URL after the proxy address https://gojdippmaps.azurewebsites.net/proxy.ashx and ? as a parameter and then adding f=json as a second parameter, preceded by &. What should happen is QGIS should recognize the proxied URL and add parameters after this using ? first and & thereafter. If I try the following request (with the referer header above) in a web browser, it is successful - note the second ?:

https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Historic_Buildings/Historic_Buildings/FeatureServer?f=json

[gioman]

---

* Right click on the ArcGIS Feature Service tool in the Browser window, give the connection a name and add the following URL

* https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Historic_Buildings/Historic_Buildings/FeatureServer

@MatzFan ?

[MatzFan]

You need the referer header, as I described. In a web browser I use an extension to add Referer: https://www.gov.je//citizen/Planning/Pages/HistoricEnvironmentDetail.aspx. The server also only accepts JSON requests, so in a web browser you must add ?f=json, as I said.

[gioman]

You need the referer header, as I described. In a web browser I use an extension to add Referer:

@MatzFan can't you just post the URL we should test in QGIS?

[MatzFan]

@gioman not sure I understand. My OP describes the issue and what I am trying to achieve. I know the server and layer on it is reachable as when I try https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Historic_Buildings/Historic_Buildings/FeatureServer?f=json with Referer: https://www.gov.je//citizen/Planning/Pages/HistoricEnvironmentDetail.aspx in a web browser I get the JSON response shown above.

If I use https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Historic_Buildings/Historic_Buildings/FeatureServer in QGIS (with the same referer) QGIS appends &?f=json not ?f=json to the request which results in a Bad Request response. Forgive me if I have not explained this adequately.

[gioman]

not sure I understand. My OP describes the issue and what I am trying to achieve. I know the server and layer on it is reachable as when I try https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Historic_Buildings/Historic_Buildings/FeatureServer?f=json

@MatzFan sorry, it is me that do not understand. Shouldn't

https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Historic_Buildings/Historic_Buildings/FeatureServer?f=json

return something meaningful rather than an error? If the above URL errors out there is no chance that it will work in QGIS, I think.

[MatzFan]

OK, sure, if I use that URL, manually including the ?f=json and the referer I connect, but the layer errors. It does this becuase QGIS tries to append the first layer index 0 to that URL; https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Historic_Buildings/Historic_Buildings/FeatureServer?f=json/0. Here is the Layer Properties window:

The correct URL for the layer is https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Historic_Buildings/Historic_Buildings/FeatureServer/0?f=json and should look like this:

[MatzFan]

My guess is the fix should identify a proxied URL of the kind in this example and simply use ? before the first parameter instead of & - I think it is that simple.

[gioman]

The correct URL for the layer is https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Historic_Buildings/Historic_Buildings/FeatureServer/0?f=json

it fails also in browser:

{"error": {"code": 403,"message":"403 - Forbidden: Access is denied.","details":["message":"Current proxy configuration settings do not allow requests which do not include a referer header."]}}

[MatzFan]

You are setting the Referer header to https://www.gov.je//citizen/Planning/Pages/HistoricEnvironmentDetail.aspx in your web browser request?

[MatzFan]

curl -H 'Referer: https://www.gov.je//citizen/Planning/Pages/HistoricEnvironmentDetail.aspx' https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Historic_Buildings/Historic_Buildings/FeatureServer/0?f=json may be easier.

[eresonance]

OK, after poking around at this I gave up and ended up making a python proxy for a proxy to get rid of the ? in the URL for QGIS. See my answer to @MatzFan's question here: https://gis.stackexchange.com/a/420806/198469

As for what's going on in QGIS, I'm not familiar with the code at all but I think URL handling for the qgsarcgisrest* files is too simplistic and assumes bizarre arcgis proxies won't break RFC3986 rules...

Take this example: https://github.com/qgis/QGIS/blob/480df9d918b323e07a61d89368f364870a94b86b/src/core/providers/arcgis/qgsarcgisrestquery.cpp#L82

QVariantMap QgsArcGisRestQueryUtils::getObjects( const QString &layerurl, const QString &authcfg, const QList<quint32>  /* snip */ )
{
  //...
  QUrl queryUrl( layerurl + "/query" );
  QUrlQuery query( queryUrl );

A QUrl is constructed by adding a dir to the end of whatever is passed in by the user, which is fine I guess. However this now-modified URL is passed into QUrlQuery. You would do this to initialize the key=value pairs that the QUrlQuery is collecting, but it's not clear to me why that's done here as the URL passed in by the user cannot have any query parameters added onto it by definition. Otherwise you can't just append a dir like what's done for the QUrl.

Then what follows is construction of the query pairs:

  query.addQueryItem( QStringLiteral( "f" ), QStringLiteral( "json" ) );
  query.addQueryItem( QStringLiteral( "objectIds" ), ids.join( QLatin1Char( ',' ) ) );
  const QString wkid = crs.indexOf( QLatin1Char( ':' ) ) >= 0 ? crs.split( ':' )[1] : QString();
  query.addQueryItem( QStringLiteral( "inSR" ), wkid );
  query.addQueryItem( QStringLiteral( "outSR" ), wkid );

The QUrlQuery treats the second maps.gov.je part of the full URL https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Historic_Buildings/Historic_Buildings/FeatureServer as the first parameter to the azure part of the URL, and simply adds the rest of the stuff to that (f, objectIds, etc.) with the & delimiter.

And finally setting the query on the original QUrl:

  queryUrl.setQuery( query );
  return queryServiceJSON( queryUrl,  authcfg, errorTitle, errorText, requestHeaders, feedback );

Obviously the first ? in the URL is screwing up everything afterwards as the second ? isn't inserted for the maps.gov.ie part of the request.

My suggestion/hack for a potential fix is to add a wrapper class for QUrl to handle this "double-?" case and to not pass the original URL into the constructor of the QUrlQuery because it's not needed and complicates query construction.

[abdullahO2]

@nyalldawson Hello how to add ArcGIS REST Server Esri/resource-proxy to QGIS