Consolidate GeoSpatial API / Service Format String

[t83714]

Consolidate GeoSpatial API / Service Format String

Related ticket: #3 Related PR: https://github.com/magda-io/magda-minion-format/pull/4

Motivation

• Better categorise the GeoSpatial API / Service Format so it's easier to select correct visualisation software.
  • e.g. TerriaJs may use different catalogItem to process different type of APIs. We need to make sure the format category can at least allow us to determine which TerriaJs catalogItem should be used to consume the API
• Consistency among projects
  • Digital Twin projects use a forked version format-minion which introduced different format categories.
  • We should make format categories more consistent among all our projects (so it's easier to handle it later)

Proposed Solution

Information used to determine format string

All information available (unless we want to query the remote unknown API) is:

• Original format metadata entered by data custodian. i.e. dcat-distribution-strings.format
• URls: i.e. dcat-distribution-strings.downloadURL or dcat-distribution-strings.accessURL

We will apply proposed logic on the information above in order to determine the actual format string

Format Detection Logic

• ESRI WMS / GeoServer WMS
  • Format String: WMS
  • TerriaJs Processing CatelogItem: WebMapServiceCatalogItem
  • Detection Logic:
  • Test URL using Regex /.*\\?.*service=wms.*/i first.
  • If not matched, test if original format metadata includes one of the following keywords:
    • "OGC:WMS-1.3.0-http-get-capabilities",
    • "OGC:WMS-1.3.0-http-get-map",
    • "OGC:WMS",
    • "OGC:WMS-1",
    • "OGC:WMS-2",
    • "OGC WMS"
• ~WMS Group~
  • ~Yet to determine how to detect WMS Group API endpoint. The URL pattern looks similar to ordinary WMS items. Might have to test the API response in order to figure it out.~
  • ~Also needs terriajs support so its catalog creation logic can recognise & trust this new format string.~
  • ~We seems already have WMS group checking logic in v7 MagdaCatalogItem~
  • We should NOT create a new "WMS Group" format. Instead, we should adopt WMS group checking logic in [v7 MagdaCatalogItem](https://github.com/magda-io/magda-preview-map/blob/master/lib/Models/MagdaCatalogItem.js#L483 at terria side. We probably should also allow user select a layer.
• ESRI WFS / GeoServer WFS
  • Format String: WFS
  • TerriaJs Processing CatelogItem: WebFeatureServiceCatalogItem
  • Detection Logic:
  • Test URL using Regex /.*\\?.*service=wfs.*/i first.
  • If not matched, test if original format metadata includes one of the following keywords:
    • "OGC-WFS"
    • "OGC:WFS-1"
    • "OGC:WFS-2"
    • "OGC WFS"
• ESRI MapServer
  • Format String: ESRI MapServer
• ESRI Feature Server
  • Format String: ESRI FeatureServer
  • when url matches FeatureServer/d (i.e. url ending with / + number), for v8 render as esri-featureServer
  • when url matches FeatureServer$|FeatureServer/$ (i.e. url ending with / and no number), for v8 render as esri-featureServer-group
• ESRI scenes Server / Service or i3s (Indexed 3D Scene Layers) service from other vendors
  • Format String: ~i3s (or ESRI SCENESERVER any suggestion? by the way, i3s is a OGC standard already)~ ESRI SceneServer
  • TerriaJs partially support via i3s to 3d tiles
  • Detection Logic: TBA
• Other ESRI REST APIs
  • Format String: ESRI REST
  • Not supported for visualisation
  • Detection Logic:
  • Any APIs that doesn't match any of those above
  • AND have original format metadata include one of the following keywords
    • "esri rest"
    • "ESRI ArcGIS"

Update July 2021

will now make both ESRI MapServer & ESRI Feature Server format as "ESRI MAPSERVER" due to here. Will create a seperate PR to terriajs and make ESRI FeatureServer show as "ESRI FeatureServer" after the PR is merged and deployed to NSW DT.

Update: release v1.0.0-rc.0 that mark ESRI Feature Server as ESRI MapServer for compatibility reason above. All future releases (i.e. v1.0.0-rc.1, v1.0.0), will all mark ESRI Feature Server as ESRI Feature Server correctly.

References & Links

• Esri REST: https://developers.arcgis.com/rest/services-reference/get-started-with-the-services-directory.htm
• i3s-spec:
  • https://github.com/Esri/i3s-spec
  • https://www.ogc.org/standards/i3s
• Terria i3s to 3d tiles
  • https://github.com/TerriaJS/i3s-to-3dtiles
• MagdaCatelogueItem

[t83714]

@mwu2018 When you have time, please have a look and let me know your thoughts on it (and whether can help to move away from the forked version)

[mwu2018]

For datasets from magda registry, the v8 version of terriajs (branch next) looks for format aspect with patterns specified by formatRegex in https://github.com/TerriaJS/terriajs/blob/da49c023055435dc495d6392507c6f7c7cdd9754/lib/Models/MagdaReference.ts#L45 As long as Magda reports the format strings in those patterns, the terriajs will recognise them.

[mwu2018]

MagdaCatelogueItem is replaced by MagdaReference in v8 terriajs.

[mwu2018]

For SceneServer, look at https://github.com/TerriaJS/magda-minion-terria/blob/master/src/onRecordFound.ts

[t83714]

closed via PR https://github.com/magda-io/magda-minion-format/pull/13