Can't load large ArcGIS feature service

[soneil1]

i get this error when trying to load a large feature service layer into qgis. Another layer that's smaller in size within the same service loads fine. Layer is not valid: The layer crs='EPSG:26918' url='https://gisservices.its.ny.gov/arcgis/rest/services/BuildingFootprints/FeatureServer/2' is not a valid layer and can not be added to the map. Reason: AFSProvider getObjectIds failed: Parsing error - illegal value

[gioman]

@soneil1 does it loads fine on other GIS software?

[soneil1]

Yes, it loads in ArcMap and ArcGIS online

[gioman]

There is also timeout error in QGIS logs

2020-03-10T14:37:15     WARNING    Network request https://gisservices.its.ny.gov/arcgis/rest/services/BuildingFootprints/FeatureServer/2/query?f=json&where=1%3D1&returnIdsOnly=true timed out
2020-03-10T14:44:17     WARNING    Host not found

that could be very well the root cause of the issue...

[haubourg]

@soneil1 can you try to raise your timeout delay in general settings / network and see if it helps?

[soneil1]

I have the timeout set to 180000

[gioman]

I have the timeout set to 180000

then maybe is just a super slow endpoint.

[Saijin-Naib]

WARNING Network request https://gisservices.its.ny.gov/arcgis/rest/services/BuildingFootprints/FeatureServer//2/query?f=json&where=1%3D1&returnIdsOnly=true timed out Interesting... Tested locally here on 3.12 and received the same Message as above, but only after an attempted connection took in excess of 30s.

Layer 1 (the availability footprint) loads just fine (near-instant). It does seem to corroborate his report.

@soneil1, as an aside, there is decent building footprint data that Microsoft provided to OpenStreetMap, and the NYS GIS Clearinghouse has that available for download as well.

[soneil1]

Yes, the availability layer works fine but it's a smaller layer. The footprints layer has 7 million records in it.

@Saijin-Naib we do have the microsoft layer, we're actually having a separate publishing issue with that one right now.

[soneil1]

Do you know if there's anything I can change on the server side?

[gioman]

Do you know if there's anything I can change on the server side?

@soneil1 use WFS? ;)

[soneil1]

@gioman we do have wfs enabled!

[Saijin-Naib]

Yes, the availability layer works fine but it's a smaller layer. The footprints layer has 7 million records in it.

@Saijin-Naib we do have the microsoft layer, we're actually having a separate publishing issue with that one right now.

Nice to e-meet you, as a consumer/resident ;)

It is quite odd that Desktop & Pro will both load/connect to the footprints instantly and QGIS chokes...

[gioman]

@gioman we do have wfs enabled!

@soneil1 URL?

[soneil1]

@Saijin-Naib I agree

[soneil1]

@gioman http://gisservices.its.ny.gov/arcgis/services/BuildingFootprints/MapServer/WFSServer?request=GetCapabilities&service=WFS

[gioman]

@gioman http://gisservices.its.ny.gov/arcgis/services/BuildingFootprints/MapServer/WFSServer?request=GetCapabilities&service=WFS

@soneil1 works but it just loads 1000 features, have you set some limit server side?

[soneil1]

@gioman yes it's set at 1000 for max number of features returned by service. we might need to up that if the feature service won't load in qgis.

[gioman]

@gioman yes it's set at 1000 for max number of features returned by service. we might need to up that if the feature service won't load in qgis.

@soneil1 if we want to understand where the issue here we need to test both services from the same server with the same full dataset.

[soneil1]

@gioman in our parameter properties our max number of records returned is set to 50,000. I change the wfs max number to 50,000 as well.

[elpaso]

Be aware of this upstream bug too: https://bugreports.qt.io/browse/QTBUG-76821

[gioman]

@gioman in our parameter properties our max number of records returned is set to 50,000. I change the wfs max number to 50,000 as well.

@soneil1 don't know.... but the service seems dead slow... in fact after adding the layer QGIS never loads any feature on map. And no, is not QGIS or my connection. I tested a different WFS server (based on Mapserver) with a layer with 6+ million features and it works ok.

[nyalldawson]

The footprints layer has 7 million records in it.

FeatureServer is a horrendously inefficient protocol for transmission of large layers like this. It's an extremely verbose text based format, so you'll be waiting for hours for this amount of data to transfer.

A potential workaround is to restrict the layer's contents to the current bounding box of the canvas by clicking this option:

That will limit the layer to only retrieving the features visible in the current bounding box.

(I think this report should be left open, because there is some work we could do to defer the initial afs provider initialisation into the feature iterator, so at least it wouldn't block the whole qgis window while this download occurs)

[PeterReyne]

Hi,

I also came across this issue a month or so ago while doing some other testing with QGIS and ArcGIS Server 10.7.1

The root cause: ArcGIS Server doesn't apply the configured query feature limit to requests that have "returnIdsOnly=true" param set and the response is quite slow for large datasets.

The potential fix: QGIS could use "resultRecordCount" and "resultRecordOffset" params to limit the response to a certain number of records (e.g. 1000) like below.

https://gisservices.its.ny.gov/arcgis/rest/services/BuildingFootprints/FeatureServer/2/query?f=json&where=1%3D1&returnIdsOnly=true&resultRecordCount=1000&resultOffset=0

an extension: QGIS could set the Query "resultRecordCount" value using the "maxRecordCount" value from this request https://gisservices.its.ny.gov/arcgis/rest/services/BuildingFootprints/FeatureServer/2?f=json

[Pedro-Murteira]

@soneil1 Still true on recent releases?

[Saijin-Naib]

Yep, seems to be:

2022-04-26T12:33:14     WARNING    Network request https://gisservices.its.ny.gov/arcgis/rest/services/BuildingFootprints/FeatureServer/3/query?f=json&where=1%3D1&returnIdsOnly=true timed out

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd">
<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8" /><style type="text/css">
p, li { white-space: pre-wrap; }
</style></head><body>

QGIS version | 3.24.2-Tisler | QGIS code branch | Release 3.24
-- | -- | -- | --
Qt version | 5.15.3
Python version | 3.9.9
GDAL/OGR version | 3.4.2
PROJ version | 8.2.1
EPSG Registry database version | v10.041 (2021-12-03)
GEOS version | 3.10.2-CAPI-1.16.0
SQLite version | 3.36.0
PDAL version | 2.4.0
PostgreSQL client version | unknown
SpatiaLite version | 5.0.1
QWT version | 6.1.5
QScintilla2 version | 2.11.6
OS version | KDE Flatpak runtime
  |   |   |  
Active Python plugins
sagaprovider | 2.12.99
db_manager | 0.1.20
processing | 2.12.99
grassprovider | 2.12.99

</body></html>QGIS version
3.24.2-Tisler
QGIS code branch
[Release 3.24](https://github.com/qgis/QGIS/tree/release-3_24)
Qt version
5.15.3
Python version
3.9.9
GDAL/OGR version
3.4.2
PROJ version
8.2.1
EPSG Registry database version
v10.041 (2021-12-03)
GEOS version
3.10.2-CAPI-1.16.0
SQLite version
3.36.0
PDAL version
2.4.0
PostgreSQL client version
unknown
SpatiaLite version
5.0.1
QWT version
6.1.5
QScintilla2 version
2.11.6
OS version
KDE Flatpak runtime

Active Python plugins
sagaprovider
2.12.99
db_manager
0.1.20
processing
2.12.99
grassprovider
2.12.99