Open mweech opened 7 years ago
moved to RCS repo.
We are getting the same issue with another layer from one of the partners. The commonality seems to be that both of these layers are coming from AGOL.
Here i s the payload that we are trying to register:
{ "version": "2.0", "fr": { "service_url": "http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0", "service_type": "esriFeature", "service_name": "Sites de camping", "metadata": { "catalogue_url": "https://gcgeo.gc.ca/geonetwork/metadata/fre/74054d44-68cf-41af-8919-5f09f80dcd02", "metadata_url": "https://gcgeo.gc.ca/geonetwork/srv/fre/xml.metadata.download?uuid=74054d44-68cf-41af-8919-5f09f80dcd02&fromWorkspace=" } }, "en": { "service_url": "http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0", "service_type": "esriFeature", "service_name": "Campsites", "metadata": { "catalogue_url": "https://gcgeo.gc.ca/geonetwork/metadata/eng/74054d44-68cf-41af-8919-5f09f80dcd02", "metadata_url": "https://gcgeo.gc.ca/geonetwork/srv/eng/xml.metadata.download?uuid=74054d44-68cf-41af-8919-5f09f80dcd02&fromWorkspace=" } } }
Here is the log info we are getting back:
2017-12-05T16:27:03.948Z {"msg": "Problem communicating with service endpoint http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0 content-type"}
2017-12-05T16:27:03.948Z put failed 400
2017-12-05T16:27:03.078Z HMAC_SHA256(msg,key): PKoX3MNIlor0okpzRfM4lnoAwj4AoS9mZ6eaL84N5qs
2017-12-05T16:27:03.072Z key: test_-k
2017-12-05T16:27:03.071Z msg: /v2/register/74054d44-68cf-41af-8919-5f09f80dcd02jstest2017-12-05T16:27:03.070Z{ "version": "2.0", "fr": { "service_url": "http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0", "service_type": "esriFeature", "service_name": "Sites de camping", "metadata": { "catalogue_url": "https://gcgeo.gc.ca/geonetwork/metadata/fre/74054d44-68cf-41af-8919-5f09f80dcd02", "metadata_url": "https://gcgeo.gc.ca/geonetwork/srv/fre/xml.metadata.download?uuid=74054d44-68cf-41af-8919-5f09f80dcd02&fromWorkspace=" } }, "en": { "service_url": "http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0", "service_type": "esriFeature", "service_name": "Campsites", "metadata": { "catalogue_url": "https://gcgeo.gc.ca/geonetwork/metadata/eng/74054d44-68cf-41af-8919-5f09f80dcd02", "metadata_url": "https://gcgeo.gc.ca/geonetwork/srv/eng/xml.metadata.download?uuid=74054d44-68cf-41af-8919-5f09f80dcd02&fromWorkspace=" } } }
2017-12-05T16:12:10.601Z {"readyState":4,"responseText":"null","responseJSON":null,"status":404,"statusText":"NOT FOUND"}
2017-12-05T16:12:10.601Z get failed 404
2017-12-05T16:12:10.320Z GET URL: /v2/doc/en/74054d44-68cf-41af-8919-5f09f80dcd02
After some investigations using the JSON that @yblain provided, I've discovered a few things
universal.py
was trying to access content-type
in r.header
which caused an error because it wasn't defined[Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
. It still requires more investigations, however it's not the main problem so will investigate it laterV2
(version 2). The code in RCS was actually creating a V1
(version 1). This seems to be the main reason why the layer failed.Here are some inputs from @james-rae about point number 3
Existing code is broken for FeatureServer, as it depends on /legend endpoint
The code using this is generating a symbology node on a final product that gets stored in RCS. This code is in make_v1_feature_node, which supports RAMP v1. You'll notice RAMP v2 does not use symbology in it's layer config
Initial Idea (needs feedback from Aly / Dan): I'm not quite sure what make_v1_feature_node is used for big-picture. I'm assuming it's if somebody does a GET in RCS for a v1 style entry, then it returns this object instead of the v2 one. I think we should consider skipping the symbology part for make_v1_feature_node if it's a FeatureServer (or just fails in general). The result, if I have it correct, would mean a v2 entry will be ok for RAMP v2, but if a version of RAMP v1 tries to consume that RCS key, it will be missing it's symbology. This will look ugly, but RAMP v1 was never promised to work for FeatureServer anyways.
After reading what @james-rae and did some testings. I have found that replacing this with esri.make_feature_node(json_request[lang])
would work. Maybe @alyec would have some inputs on it? :package:
The error with creating a v1 node was caused by requesting the legend of the layer from the server which it did not exit (link). It seems commenting out node['symbology'] would also work.
There are 2 solutions I can think of as for now
make_feature_node
instead of make_v1_feature_node
(i.e. make a v2 node instead of a v1 node)Map Server
or a Feature Server
. If it is coming from Feature Server
, then RCS would avoid running node['symbology']Personally I think the second option is better, simply because we will still be able get a v1 style entry using GET
.
A third option, which might prevent a number of errors in RAMP v1
, is if a Feature Server is incoming, the RCS v1
code inserts a placeholder in the symbology
node (e.g. A question-mark graphic and the word "Unknown" for text).
But before doing this, I think we need to decide if RAMP v1
requesting an RCS key that refers to a Feature Server is a valid use case. If not, is easier just to skip the generation of RCS v1
entries for Feature Server (or something similar).
@james-rae you have a good point re: we probably would never run into v1 entries for Feature Server being requested unless we get an odd configuration mismatch of DC and RCS/RAMP.
There are 3 issues related to Metadata/Catalog that I have identified using the JSON @yblain provided
The certificates failed when requesting from metadata/catalog URLs. However I was able to bypassed this issue with setting verify=False
in link but it should not be a permenant solution as it exposes vulnerability. Is this issue due to running from Vagrant
(was running from Vagrant
) or could it have something to do with the security setting on the server?
{ "msg": "Metadata URL: \"https://gcgeo.gc.ca/geonetwork/srv/eng/xml.metadata.download?uuid=74054d44-68cf-41af-8919-5f09f80dcd02&fromWorkspace=\" request failed with error \"[Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed\"" }
{ "msg": "Catalogue URL: \"https://gcgeo.gc.ca/geonetwork/metadata/eng/74054d44-68cf-41af-8919-5f09f80dcd02\" request failed with error \"[Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed\"" }
The MIME type of requested data from the provided metadata enpoint was not what RCS expected it would be. RCS was expecting the MIME type to be xml
; what it got back was octet-stream
. Should we be supporting octet-stream, charset=UTF-8
?
{ "msg": "Metadata URL: \"https://gcgeo.gc.ca/geonetwork/srv/eng/xml.metadata.download?uuid=74054d44-68cf-41af-8919-5f09f80dcd02&fromWorkspace=\" could not be retrieved: expected \"['application/xml', 'text/xml']\", got \"application/octet-stream;charset=UTF-8\"" }
The request from the provided Catalog URL failed. A couple things to point out. The link returned a valid page when pasted in the browser, and the MIME type what RCS expected was html
from a Catalog endpoint.
{ "msg": "Catalogue URL: \"https://gcgeo.gc.ca/geonetwork/metadata/eng/74054d44-68cf-41af-8919-5f09f80dcd02\" could not be retrieved: bad HTTP status code received 404" }
Probably a stale CA store, CA certs probably need to be upgraded. Destroying and recreating the vagrant env should help in Barry's case, if not maybe some change to the Vagrant config is in order. For production we can sort out in a separate issue, but my first question will be if the server is fully patched.
Rather than expand the list to include octet-stream
I'd rather put in a config option to drop the check if we know a server isn't going to give back useful data via MIME types. octet-stream
is far too general to draw conclusions from IMO and I don't think it really gives us any additional confidence that the viewer isn't going to have issues accessing the service.
I spent a bit more time on this than I probably should have. There's some really odd behaviour and it seems the Accept
header is the culprit.
This fails:
curl -v --http1.1 -i "https://gcgeo.gc.ca/geonetwork/metadata/eng/74054d44-68cf-41af-8919-5f09f80dcd02"
but this works:
curl -v --http1.1 -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -i "https://gcgeo.gc.ca/geonetwork/met adata/eng/74054d44-68cf-41af-8919-5f09f80dcd02"
In the first case the header goes out as Accept: */*
which I can't see being objectionable to the server, but I guess it is. I think we should try to figure out a bit more of what's going on within the server.
So is this closed?
Upgraded to v2.3.0 and still can't register:
This service adds fine on our test RCS instances:
{"fr":{"service_url":"http://services1.arcgis.com/HsjBaDykC1mjhXz9/ArcGIS/rest/services/FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705/FeatureServer/0","service_type":"esriFeature","service_name":"Flood Extent FR"},"en":{"service_url":"http://services1.arcgis.com/HsjBaDykC1mjhXz9/ArcGIS/rest/services/FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705/FeatureServer/0","service_type":"esriFeature","service_name":"Flood Extent"},"version":"2.0"}
http://fgpv.cloudapp.net/demo/develop/prod/samples/index-fgp-en.html?keys=floodextent
I did't have any troubles registering the camp sites layer.
Here is the snippet:
{
"version":"2.0",
"fr":{
"service_url":"http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0",
"service_type":"esriFeature",
"service_name":"Sites de camping"
},
"en":{
"service_url":"http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0",
"service_type":"esriFeature",
"service_name":"Campsites"
}
}
@jvanulde What did the error message say?
2018-08-24T19:16:53.753Z {"msg": "Problem communicating with service endpoint http://services2.arcgis.com/wCOMu5IS7YdSyPNx/arcgis/rest/services/campsites_sites_camping_vw/FeatureServer/0 No JSON object could be decoded"}
2018-08-24T19:16:53.753Z put failed 400
Still an issue. See log: http://redmine.gcgeo.gc.ca/redmine/attachments/download/5481/rcs.log where this error is provided:
FeatureServer registration must specify a feature layer
@yblain commented on Tue Jun 20 2017
A bug has been identified with regards to the registration of a service in RCS.
Here is the description of the problem from Jose's point of view:
Hi
I have been reported a problem with registering the following layer in RCS:
http://services1.arcgis.com/HsjBaDykC1mjhXz9/ArcGIS/rest/services/FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705/FeatureServer/0
The Data Catalogue is sending the following request:
{ "en": { "service_url": "http://services1.arcgis.com/HsjBaDykC1mjhXz9/ArcGIS/rest/services/FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705/FeatureServer/0", "service_type": "esriFeature", "service_name": "FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705" }, "fr": { "service_url": "http://services1.arcgis.com/HsjBaDykC1mjhXz9/ArcGIS/rest/services/FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705/FeatureServer/0", "service_type": "esriFeature", "service_name": "FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705" }, "version": "2.0" }
But getting the following error:
{ "msg": "Problem communicating with service endpoint http://services1.arcgis.com/HsjBaDykC1mjhXz9/ArcGIS/rest/services/FloodExtentPolygon_SouthManitoba_MODIS_20170405_1530_1705/FeatureServer/0 content-type" }
Tested also with the RCS test page and same issue.
The request looks fine, but maybe not sure if maybe the service_type value is not what is expecting RCS?
If you can provide me some feedback on how to manage this is really appreciated, thanks.
The related issue in Redmine is this one: http://redmine.gcgeo.gc.ca/redmine/issues/9156
Regards, Jose García