Add support for loading remote resources

[ricardogsilva]

GeoNode is able to show layers from remote services.

It currently supports loading layers from either WMS services or ESRI ArcGIS REST. Lets add initial support for WMS remote services. The current version of V2 API does not include the service URLs in the API responses, so for now lets try to come up with a workaround. We can revisit later and add proper support.

Layers that come from remote services have their storeType property set to remoteStore, as can be seen below (only relevant properties are shown):

https://master.demo.geonode.org/api/v2/layers/759

{
    "layer": {
        "pk": "759",
        "uuid": "268895d3-2c0b-4903-850a-7b721ff2ca69",
        "name": "ne_110m_admin_0_countries_lakes",
        "workspace": "remoteWorkspace",
        "store": "httpsdevgeonodegeo-solutionsitgsows",
        "storeType": "remoteStore",
        "charset": "UTF-8",
        "resource_type": "layer",
        "polymorphic_ctype_id": "58",
        "title": "ne_110m_admin_0_countries_lakes",
        "abstract": "No abstract provided",
        "attribution": null,
        "doi": null,
        "alternate": "geonode:ne_110m_admin_0_countries_lakes",
        "srid": "4326",
        "detail_url": "/layers/httpsdevgeonodegeo-solutionsitgsows:geonode:ne_110m_admin_0_countries_lakes",
        "spatial_representation_type": null
    }
}

It seems we can even determine the type of remote service by looking at the store property.

[ricardogsilva]

Upon closer inspection it seems there is no reliable way to infer the URL for a remote service from the current V2 API. As such, let's wait until geosolutions-it/nexus-geonode#22 is fixed upstream in order to implement this with the V2 API.

In the meanwhile you may move forward and PR an implementation for the CSW API client. This one shall include the proper link to the remote service. Ok @Samweli ?

[ricardogsilva]

@Samweli any progress on this?

As said above, we should be able to add support for the CSW API at least.

[Samweli]

@ricardogsilva I'm taking a look at it now, will get back to you.

[Samweli]

@ricardogsilva looking at the CSW record, I couldn't find a way to indicate if a record is from remote source using the gmd:contentInfo tag, it is blank for remote resources, also the available online links in the record doesn't point to the remote source. We can only get the information about the record remote resources inside the GeoNode API V1 layer details.

[Samweli]

@ricardogsilva seems we can do this by using the same approach in loading layers from CSW API.

[ricardogsilva]

With the current status of the GeoNode it is possible to distinguish between local and remote resources. There are seemingly some hurdles to overcome on the GeoNode side though. Paraphrasing my own comment from geosolutions-it/nexus-geonode#508:

However, support for remote resources in the QGIS plugin is currently not really working due to, IMO, some GeoNode bugs.

The current version of the plugin only loads datasets whose subtype property is either vector or raster. It does not really distinguish if datasets are local or remote for now (although we could implement something like that by looking at the API response's sourcetype field, which shows a value of REMOTE for remote resources).

Regardless of the dataset origin, if the API response includes a links list with each object having keys with link_type: "OGC:WMS", link_type: "OGC_WFS" or link_type: "OGC_WCS" then the plugin will offer buttons for loading the dataset onto QGIS map canvas.

However, from a brief investigation, it seems there are two different implementations of remote resources in GeoNode at the moment. When querying master.demo.geonode.org for a list of remote datasets, like this:

https://master.demo.geonode.org/api/v2/datasets/?filter{sourcetype}=REMOTE

The returned datasets have subtype=remote. To me this seems like a bug in GeoNode, as the same datasets already have sourcetype=REMOTE, and also I assume the subtype property is meant for specifying the subtype of a dataset, not its origin. Therefore, as mentioned above, these resources are not recognized by the plugin and thus do not show up on search results. Presumably, these datasets have been generated with GeoNode's remote services feature.

The other implementation would be the one where datasets come straight from the geonode.harvesting module. I'm not sure, but I am assuming that master.demo.geonode.org does not have any such datasets at the moment. The NEXUS staging environment also does not seem to contain any remote datasets at the moment, as issuing this request

https://dev.nexus.geosolutionsgroup.com/api/v2/datasets/?filter{sourcetype}=REMOTE

returns a total of 0 results.

On my own local dev instance, which is up to date with the NEXUS repo master branch, I have remote resources which were generated by using a harvester. When issuing a request like:

http://localhost/api/v2/datasets/?filter{sourcetype}=REMOTE

The resulting datasets do have subtype=vector. This means that datasets coming straight from the harvesting module do not suffer from the aforementioned bug. This means the QGIS plugin is able to recognize them and show them on the datasets list. Unfortunately, the API response does not include any links.link property with a suitable link_type, which means that the QGIS plugin is not able to offer to load them as layers in QGIS, since it does not know how to get them. To me this seems like a different bug in GeoNode, whereby the links property of the API response is incomplete.

Note that, when requesting a dataset's detail page, with a request such as

http://localhost/api/v2/datasets/{datasetId}/

the returned dataset does have a ows_url property, which can be used to construct the relevant OWS requests. This means that GeoNode does have enough information to build a correct links property on the dataset list endpoint.

In conclusion, it seems to me that remote resources should now be working OK on the QGIS plugin but they are currently broken due to some inconsistencies in the GeoNode API.