Implement Support for 'Recursive' Registration of Multi Layer Services

[mweech]

Need to implement "recursive" mode for registration of layers for both ogcWMS and esriMapServices.

[alyec]

Need to define how to access recursively scraped layers. The initial concept was to have different endpoints (e.g. something like /group_layer/path_to_sublayer this will require coordination with the viewer and likely a change in the viewer schema).

[mweech]

Example for OGC WMS: http://maps.geogratis.gc.ca/wms/railway_en?SERVICE=WMS&REQUEST=GetCapabilities

Registration would be done by specifying the URL "http://maps.geogratis.gc.ca/wms/railway_en?SERVICE=WMS&REQUEST=GetCapabilities" and "recursive" mode. This would lead to the viewer providing an entry in the legend for the parent service and then have all child layers populated (railway.structure.line, railway.structure.point, railway.track, railway.ferry, railway.subdivision, railway.station, railway.marker_post, railway.crossings).

[mweech]

Example #1 for Esri Map Service: http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer/0

Registration would be done by specifying the URL "http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer/0" and "recursive" mode. This would lead to the viewer providing an entry in the legend for the parent service and then have all child layers populated (http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer/1, http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer/2, http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer/3, http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer/4).

Example #2 for Esri Map Service: http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer

Registration would be done by specifying the URL "http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer" and "recursive" mode. This would lead to the viewer providing an entry in the legend for the parent service and then have all child layers populated (http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer/0 through 52).

[alyec]

Example 2: do you want 52 layers on the map?

WMSes can also provide aggregate layers, ex:

Canada Roads (aggregate of all provinces)
-- AB Roads
-- BC Roads
...

If you output layerEntries: ['canada_roads','ab_roads','bc_roads',...] that can lead to a fair bit of duplication and odd behaviour in the viewer.

[mweech]

re: example 2.. Ideally no. That's just an example of what would happen and there are services composed of fewer layers that can be used to test with that would be a more realistic example. The point is the model allows for a data provider to provide groups of layers under the same metadata record.

re: WMS. Again thats a data provider decision. We shouldn't exclude abilities because there are situations where its not the most desirable outcome. The provider should then decide if they want to provide a single record for all layers, or is it better to provide them as separate data sets for each province. Or, they need to rethink how/what they share within the service.

[alyec]

I'm not suggesting we do not implement recursive. Only that we add the ability to signal the viewer that these records were registered in bulk. Instead of sending [0,1,...,52] we can expose an endpoint which lets the viewer consume sublayers in a more sensible manner.

If a provider knows how they want to group layers it makes more sense for them to use the existing scrape_only option since they can specify exactly how things should be grouped. If a provider doesn't know how they want things grouped (i.e. they do not set scrape_only and do set recursive: true) then we can add an additional data on /doc/en/group_layer which indicates that sublayers can be pulled individually from /group_layer/path_to_sublayer.

[dan-bowerman]

Question, and need a decision with regards to https://github.com/fgpv-vpgf/fgpv-vpgf/issues/806.

Here's a Gitter conversation I had with Aly about this:

Dan Bowerman @dan-bowerman Jun 21 14:59 Just updated the Azure RCS so I could test with the develop auto-build version. It sort of loads, you can see the layer in the Legend at least, but nothing is displaying. http://fgpv.cloudapp.net/demo/develop/index-fgp-en.html?keys=0 It's trying to make this call: http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer/0/export?dpi=96&transparent=true&format=png8&bbox=-406697.7230971126%2C-311918.55201718735%2C2334391.092413852%2C1534876.8082402&bboxSR=3978&imageSR=3978&size=1036%2C698&f=image but of course, we're not trying to export an image from layer 0, we're trying to look at sublayers 1, 2, 3, and 4. Guess I'll need to log this somewhere.

Aly Merchant @alyec Jun 21 15:03 Right, the URL is wrong for the layer. I don't know if RCS should try to detect potentially bad service URLs

Aly Merchant @alyec Jun 21 15:17 http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer/0 is the wrong URL for a dynamic service. It should be: http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer If I try to register without the /0 it gives a 500 error.

Dan Bowerman @dan-bowerman Jun 21 15:21 But my understanding is the use case is for someone to take a service like http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer, and just register those layers under the "Nature" group This was something GeoCat brought to our attention yesterday morning.

Aly Merchant @alyec Jun 21 15:22 We haven't really defined how recursive should be used, but I would think that would be: scrape_only: [0], recursive: true. My thoughts are that if scrape_only is present then all child nodes under anything in the list will be taken similar to how legend works in the viewer: https://github.com/fgpv-vpgf/fgpv-vpgf/wiki/Config-Schema#dynamic-layers

Dan Bowerman @dan-bowerman Jun 21 15:25 OK, makes sense. I'll have to do some tweaking in the morning.

My question is: How should we handle the recursive parameter in RCS registration? Should we specify which layer has sublayers underneath with scrape_only AND recursive as Aly suggests, or should we be able to put http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer/0 and have the sub-layers underneath registered without specifying scrape_only?

If the former, scrape_only already does this, you don't need the recursive parameter. If the latter, we will likely need changes to the Viewer so it does not try to display the "0" layer which only contains sub-layers and no data.

[dan-bowerman]

versus

[mweech]

This comes back to what would be identified in a metadata record. As of right now, I am not aware of how a metadata record would be able to describe anything that was not based on an absolute URL endpoint. So for example: http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer/0 or http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer would be specified.

Any implementation that requires explicit specification of child layers requires extra information to be conveyed at a catalogue registration level, which will not be a universally performed activity if implemented.

[dan-bowerman]

{
    "fr": {
        "service_url": "http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer",
        "scrape_only": [0],
        "recursive": true,
        "service_type": "esriMapServer",
        "service_name": "CESI: Protected areas, Canada"
    },
    "en": {
        "service_url": "http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer",
        "scrape_only": [0],
        "recursive": true,
        "service_type": "esriMapServer",
        "service_name": "CESI: Protected areas, Canada"
    },
    "version": "2.0"
}

versus

{
    "fr": {
        "service_url": "http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer/0",
        "recursive": true,
        "service_type": "esriMapServer",
        "service_name": "CESI: Protected areas, Canada"
    },
    "en": {
        "service_url": "http://ec.gc.ca/arcgis/rest/services/CESI_FGP_All_Layers/MapServer/0",
        "recursive": true,
        "service_type": "esriMapServer",
        "service_name": "CESI: Protected areas, Canada"
    },
    "version": "2.0"
}

[dan-bowerman]

:heavy_check_mark: Decision Mike, James, and I concluded:

RCS will handle this. If the layer is identified as a Group Layer, drop the layer integer from the registered service URL and scrape all layers underneath.