ArcGIS Interoperability

tdipisa commented 8 months ago

Description

It is requested to improve on the interoperability to support better ArcGIS services in MapStore. This issue involves two main activities for which the following task is in common:

subtasks

Support of tokens for private layers - MapStore token support

ArcGIS WMS Server interoperability for GetFeatureInfo requests

Thanks to https://github.com/geosolutions-it/MapStore2/pull/9048, the serverType property has been recently added in MapStore to define which type of server is providing the WMS service. It should be possible to use this option to exclude the exceptions parameter not supported in an ArcGIS WMS service. During the development phase it is requested to evaluate if it is necessary an additional serverType or the NO_VENDOR type used also by other services is enough. Possibly also hide some functionalities not supported by WMS endpoint that does not support GetFeatureInfo application/json responses with geometry, to do so it is necessary also to store the information related to the list of supported get feature info responses.

Some notes about findings from the preliminary investigation:

It seems that the DescribeLayer request used by MapStore to add the search parameter with WFS functionality is not supported in the ArcGIS WMS service. This means is not possible to associate a WFS service with a WMS programmatically in addition some tests done on agsnotfed2/rest/services/Test/country_4m/MapServer show that the WMS/WFS capabilities endpoint used two different name for the layer (0/country_4m:apgispaudw.common_data.country_4m) and this makes even difficult to associate them manually (but maybe this is only a configuration problem).

Involved sub tasks are: subtasks

Review WMS get feature info request based on server type Hide the highlight feature icon of get feature info plugin if GeoJSON response is not supported. This include work related to the storing of additional information in the layer config or alternative strategy

ArcGIS MapServer Catalog and Layer support

The work for this task focus on the support of ArcGIS MapServer endpoints inside MapStore. The task should include following improvement inside MapStore:

New catalog type for ArcGIS MapServer endpoint with the possibility to list layers and add them to the map
New ArcGIS MapServer layer type implementations for OpenLayers, Cesium and Leaflet with the default lifecycle
Support of base layer settings: opacity, visibility, title, description, group and visibility limits
Support of get feature info using the /query endpoint

Some notes about findings from the preliminary investigation:

A MapServer endpoint could provide a single layer or multiple layer that could compose a layer group. It is requested to evaluate in the development phase if the information about subLayerIds and parentLayerId have effect on the visualization in map
It shoud lbe possible to use maxScale and minScale properties of a layer to set the visibility limits. In general we should take a look to all the property available in the layer object
It is possible to send a request to /MapServer/{layerId} to get additional information about the specific layer
It is possible to use the query endpoint /MapServer/{layerId}/query (POST) for Get Feature Info

Involved sub tasks are: subtasks

Support of ArcGIS MapServer endpoint in MapStore: catalog, layer visualization, get feature info and base layer settings

Acceptance criteria

[ ] It will be possible to successfully perform GetFeatureInfo requests according to the server type and show Identify results to end users
[ ] It will be possible to support layers from ArcGIS MapServer added to the map through the Catalog tool
[ ] The MS developer guide will be updated accordingly by including all related and necessary info

Other useful information

Here a branch where a base implementation sa been tested during the preliminary evaluation of this work. That should be completely refactored to be accepted in MapStore, take this just as a prototype.

Igi-ID commented 6 months ago

These two public services have been used during development of the arcgis support so far:

More specifically:

These can also be used for easier testing when development is concluded.

Igi-ID commented 6 months ago

The current state of development is the following (updating this post as progress is made):

subtasks Support of tokens for private layers - MapStore token support

[ ] Not yet implemented

WMS interoperability and GFI support

subtasks Review WMS get feature info request based on server type Hide the highlight feature icon of get feature info plugin if GeoJSON response is not supported. This include work related to the storing of additional information in the layer config or alternative strategy

[x] GetFeatureInfo requests are supported by previous work done in https://github.com/geosolutions-it/MapStore2/issues/9564, mainly due to esri servers using the geo+json mime type for these requests rather than json.
[x] As a corollary, the MS features which aid GFI requests, such as the highlight feature, should work as expected.

ArcGIS MapServer Catalog and Layer support

subtasks Support of ArcGIS MapServer endpoint in MapStore: catalog, layer visualization, get feature info and base layer settings

[x] Service support (catalog) is available (*minor issues still present)
[x] OL layer visualization is available (*minor issues still present)
[ ] Cesium layer visualization is not availble
[ ] Leaflet layer visualization is not available
[ ] GFI requests are available but fairly unstable (see minor issues below).
[x] Base layer settings are available.

Specific issues:

[ ] Catalog search tool (which filters available records) not implemented yet.
[ ] documentation and tests across the whole implementation, as well as updating our developer guides.
[ ] raster sets with an empty layers property are not currently handled (due to data structure, at the base in fetchData). These are usually background images, orthophotos and the like.
[ ] the format of the identify feature is not handled (currently hardcoded), meaning you cannot change the format from the global settings, nor layer settings (menu unavailable).
[ ] identify feature intersects tile with geometry (bbox?) rather than feature geometry with point. As a consequence, you can click even outside geometries, but inside the bounding box of a tile and get a successful request, highlighting all geometries within that tile. Metadata is aggregated in the identify panel.
[ ] Identify feature may occasionally fail by not registering the layer at all, especially in the case of spatially overlapping layers.
[ ] Updating OL layers is not handled to its full extent and there is currently a critical failure related to reprojection.
[ ] Bounding box as well as few other relevant information points are not part of the layer object.
[ ] Zoom to feature (identify panel) is not working.
[ ] Add layer feature in layer properties panel is not working.

allyoucanmap commented 6 months ago

@tdipisa @Igi-ID Some feedback:

Cesium layer visualization is not availble Leaflet layer visualization is not available

just pushed some initial code to support arcgis mapserver layer on Cesium and Leaflet see https://github.com/geosolutions-it/MapStore2/commit/92eadf0b2e68c4b4d61e163d99eb1a93591dd58a on dev_10040 so please push the code locally. The Leaflet implementation is not currently supporting tiling but for the moment we could skip this but we should ensure the Cesium implementation is working properly.

Catalog search tool (which filters available records) not implemented yet.

If the mapserver does not support pagination we should use a similar approach to the WMS one where we simulate the pagination and filter client side

raster sets with an empty layers property are not currently handled (due to data structure, at the base in fetchData). These are usually background images, orthophotos and the like.

Do you have an url example? Probably this case need to use a sort of TMS approach, looking at the Cesium code it is possible to request cached tiles stored in the tileInfo property. We should verify if it's the case.

the format of the identify feature is not handled (currently hardcoded), meaning you cannot change the format from the global settings, nor layer settings (menu unavailable).

I think it's fine for the moment to prevent the selection of the identify but I would like to understand where it has been hardcoded, could you please share the hardcoded part?

identify feature intersects tile with geometry (bbox?) rather than feature geometry with point. As a consequence, you can click even outside geometries, but inside the bounding box of a tile and get a successful request, highlighting all geometries within that tile. Metadata is aggregated in the identify panel.

Did you try to use a point geometry without reconstruct a small bbox based on scale level?

I'm seeing this implementation with an implementation similar to WMS but probably it's not needed

maybe we could use lat and long from center property and use the supported geometry point available in MapServer query

Identify feature may occasionally fail by not registering the layer at all, especially in the case of spatially overlapping layers.

Could you provide steps to reproduce this?

Updating OL layers is not handled to its full extent and there is currently a critical failure related to reprojection.

Could you double check if this has been solved with my new commit?

I think this is not needed because we are not rendering features client side, the source is based on image tiles and not vector

Bounding box as well as few other relevant information points are not part of the layer object.

I think it's missing an additionl request to the layer id endpoint to get layer metadata, we can place this request in the addLayerAndDescribeEpic. This action is triggered when the layer is added from catalog so we can add an if statement for arcgis with the request to layer.url/layer.name. This endpoint should contain all the needed info for bounding box and probably also if the layer is queryable or not, at the moment we expect at minimum the following information for the layer:

{
    type: 'arcgis',
    url: '/rest/MapServer',
    name: '0',
    visibility: true,
    opacity: 1,
    bbox: { ... }
}

Note: please does not pass the id of the layer so mapstore is able to create a uuid

Zoom to feature (identify panel) is not working.

Zoom to feature is not working because is missing the bbox property in the layer options

Add layer feature in layer properties panel is not working.

Could you explain this issue with replicable steps?

Igi-ID commented 6 months ago

I'll update this post when some answers are better formed.

@tdipisa @Igi-ID Some feedback:

Cesium layer visualization is not availble Leaflet layer visualization is not available

just pushed some initial code to support arcgis mapserver layer on Cesium and Leaflet see 92eadf0 on dev_10040 so please push the code locally. The Leaflet implementation is not currently supporting tiling but for the moment we could skip this but we should ensure the Cesium implementation is working properly.

Catalog search tool (which filters available records) not implemented yet.

If the mapserver does not support pagination we should use a similar approach to the WMS one where we simulate the pagination and filter client side

Noted on both points above. Thank you for bringing up server side pagination, I'll re-check that and proceed accordingly.

raster sets with an empty layers property are not currently handled (due to data structure, at the base in fetchData). These are usually background images, orthophotos and the like.

Do you have an url example? Probably this case need to use a sort of TMS approach, looking at the Cesium code it is possible to request cached tiles stored in the tileInfo property. We should verify if it's the case.

http://www.pcn.minambiente.it/arcgis/rest/services/immagini/IGM_25000/MapServer

the format of the identify feature is not handled (currently hardcoded), meaning you cannot change the format from the global settings, nor layer settings (menu unavailable).

I think it's fine for the moment to prevent the selection of the identify but I would like to understand where it has been hardcoded, could you please share the hardcoded part?

https://github.com/geosolutions-it/MapStore2/blob/92eadf0b2e68c4b4d61e163d99eb1a93591dd58a/web/client/utils/mapinfo/arcgis.js#L54

identify feature intersects tile with geometry (bbox?) rather than feature geometry with point. As a consequence, you can click even outside geometries, but inside the bounding box of a tile and get a successful request, highlighting all geometries within that tile. Metadata is aggregated in the identify panel.

Did you try to use a point geometry without reconstruct a small bbox based on scale level?

Good pointer, will retest

I'm seeing this implementation with an implementation similar to WMS but probably it's not needed

maybe we could use lat and long from center property and use the supported geometry point available in MapServer query

Identify feature may occasionally fail by not registering the layer at all, especially in the case of spatially overlapping layers.

Could you provide steps to reproduce this?

My report on the issue is incomplete and badly phrased. It's essentially this reported bug: https://github.com/geosolutions-it/MapStore2/issues/10227

namely, when we put layers into a group and select the group in the panel, GFI will resolve with 'No active queryable layer'.

Updating OL layers is not handled to its full extent and there is currently a critical failure related to reprojection.

Could you double check if this has been solved with my new commit?

Solved indeed.

I think this is not needed because we are not rendering features client side, the source is based on image tiles and not vector

Bounding box as well as few other relevant information points are not part of the layer object.

I think it's missing an additionl request to the layer id endpoint to get layer metadata, we can place this request in the addLayerAndDescribeEpic. This action is triggered when the layer is added from catalog so we can add an if statement for arcgis with the request to layer.url/layer.name. This endpoint should contain all the needed info for bounding box and probably also if the layer is queryable or not, at the moment we expect at minimum the following information for the layer:
{
    type: 'arcgis',
    url: '/rest/MapServer',
    name: '0',
    visibility: true,
    opacity: 1,
    bbox: { ... }
}
Note: please does not pass the id of the layer so mapstore is able to create a uuid

Great pointer, making the adjustments.

Zoom to feature (identify panel) is not working.

Zoom to feature is not working because is missing the bbox property in the layer options

So related to the point above as expected, will retest when bbox is available

Add layer feature in layer properties panel is not working.

Could you explain this issue with replicable steps?

This is also a case of bad phrasing, the issue is related to the add layer to selected group action and the scenario is the following:

create a group, open toc, select group
click `add layer to selected group button, pick a layer from the catalog and add it to the map.
the layer is not added to the group Initial investigation showed that this action works only for WMS. Video below.

https://github.com/geosolutions-it/MapStore2/assets/90094775/a7207a6a-de05-445e-8d40-367dc13e6e7f

tdipisa commented 5 months ago

These two public services have been used during development of the arcgis support so far:

http://www.pcn.minambiente.it/arcgis/rest/services

https://hazards.fema.gov/arcgis/rest/services

More specifically:

https://hazards.fema.gov/arcgis/rest/services/public/NFHL/MapServer

https://hazards.fema.gov/arcgis/rest/services/Studies/StudiesTracking/MapServer

http://www.pcn.minambiente.it/arcgis/rest/services/ADB/MapServer

http://www.pcn.minambiente.it/arcgis/rest/services/qu_lidar_0202/MapServer

http://www.pcn.minambiente.it/arcgis/rest/services/immagini/IGM_25000/MapServer

These can also be used for easier testing when development is concluded.

This also is another one available for testing: https://sampleserver6.arcgisonline.com/arcgis/rest/services

geosolutions-it / MapStore2