Closed steffilazerte closed 8 months ago
@steffilazerte I am getting the exact same error. I have not been able to query geodata since yesterday morning.
Interesting that it's so recent! I was trying this yesterday afternoon for the first time since March, so thought it might have been a longer term issue. Perhaps it's a BC data portal issue?
Thanks for reporting. This is definitely brand new, and smells like a catalogue/wfs issue. I'll look into this today
I made a change yesterday to start caching the 'base' (all layers) WMS and WFS documents, when a specific layer is not requested. e.g. https://openmaps.gov.bc.ca/geo/pub/wfs?request=GetCapabilities but not https://openmaps.gov.bc.ca/geo/pub/WHSE_FOREST_VEGETATION.VEG_COMP_LYR_R1_POLY/wfs?request=GetCapabilities.
If a cached document is hit, you'll get a response header X-Get-Capabilties-Cache: hit
All versions (1.0.0, 1.1.0, or 2.0.0) are cached, and as before, 2.0.0 is the default if unspecified.
The cached documents should only be served to requests that wanted the base document. One URL I added to redirect to the cached GetCap doc which might be causing issues is https://openmaps.gov.bc.ca/geo/pub/wfs ?
Currently the server is not accepting POST
requests. GET
requests work, but cannot perform all of the queries we need to, so we need POST
curl -X POST -F "SERVICE=WFS" -F "VERSION=2.0.0" -F "REQUEST=DescribeFeatureType" -F "typeNames=WHSE_ENVIRONMENTAL_MONITORING.EMS_MONITORING_LOCN_GROUPS_SVW" https://openmaps.gov.bc.ca/geo/pub/wfs
Returns the entire GetCapabilities
response, not the Feature Description we are asking for (not posting here because it's too long).
GET
works:
curl https://openmaps.gov.bc.ca/geo/pub/wfs\?SERVICE\=WFS\&VERSION\=2.0.0\&REQUEST\=DescribeFeatureType\&typeNames\=WHSE_ENVIRONMENTAL_MONITORING.EMS_MONITORING_LOCN_GROUPS_SVW
<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:pub="http://delivery.openmaps.gov.bc.ca/geo/" xmlns:wfs="http://www.opengis.net/wfs/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" targetNamespace="http://delivery.openmaps.gov.bc.ca/geo/">
<xsd:import namespace="http://www.opengis.net/gml/3.2" schemaLocation="http://openmaps.gov.bc.ca/geo/schemas/gml/3.2.1/gml.xsd"/>
<xsd:complexType name="WHSE_ENVIRONMENTAL_MONITORING.EMS_MONITORING_LOCN_GROUPS_SVWType">
<xsd:complexContent>
<xsd:extension base="gml:AbstractFeatureType">
<xsd:sequence>
<xsd:element maxOccurs="1" minOccurs="1" name="MONITORING_GROUP_ID" nillable="false" type="xsd:decimal"/>
<xsd:element maxOccurs="1" minOccurs="0" name="DESCRIPTION" nillable="true" type="xsd:string"/>
<xsd:element maxOccurs="1" minOccurs="1" name="GROUP_NAME" nillable="false" type="xsd:string"/>
<xsd:element maxOccurs="1" minOccurs="1" name="OFFICE_CD" nillable="false" type="xsd:string"/>
<xsd:element maxOccurs="1" minOccurs="1" name="AEMS_TAG" nillable="false" type="xsd:string"/>
<xsd:element maxOccurs="1" minOccurs="1" name="FEATURE_CODE" nillable="false" type="xsd:string"/>
<xsd:element maxOccurs="1" minOccurs="0" name="GEOMETRY" nillable="true" type="gml:GeometryPropertyType"/>
<xsd:element maxOccurs="1" minOccurs="0" name="OBJECTID" nillable="true" type="xsd:decimal"/>
<xsd:element maxOccurs="1" minOccurs="0" name="SE_ANNO_CAD_DATA" nillable="true" type="xsd:hexBinary"/>
</xsd:sequence>
</xsd:extension>
</xsd:complexContent>
</xsd:complexType>
<xsd:element name="WHSE_ENVIRONMENTAL_MONITORING.EMS_MONITORING_LOCN_GROUPS_SVW" substitutionGroup="gml:AbstractFeature" type="pub:WHSE_ENVIRONMENTAL_MONITORING.EMS_MONITORING_LOCN_GROUPS_SVWType"/>
</xsd:schema>
Thanks @rustyjux - that could be related? As I just posted (at the same time as you I think!), all POST
requests are returning the GetCapabilities
response, which may be the cached one?
Yep that's got to be it. I tried excluding requests which include WHSE or REG in the query string but running that curl request I still get the full document.
I will try and exclude all POST requests
I think that could work, I can't think off the top of my head where we're sending queries without a layer name that looks like that... OR excluding all POSTs might be more robust. Or all POST requests that contain a query string at all?
Happy to test any/all of the potential solutions on delivery. Just let me know when to try
I tried rolling back the caching entirely on delivery. Using @ateucher's request:
curl -X POST -F "SERVICE=WFS" -F "VERSION=2.0.0" -F "REQUEST=DescribeFeatureType" -F "typeNames=WHSE_ENVIRONMENTAL_MONITORING.EMS_MONITORING_LOCN_GROUPS_SVW" https://delivery.openmaps.gov.bc.ca/geo/pub/wfs
I get this response:
<?xml version="1.0" encoding="UTF-8"?><ows:ExceptionReport xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:ows="http://www.opengis.net/ows" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0.0" xsi:schemaLocation="http://www.opengis.net/ows https://delivery.openmaps.gov.bc.ca/geo/schemas/ows/1.0.0/owsExceptionReport.xsd">
<ows:Exception exceptionCode="NoApplicableCode">
<ows:ExceptionText>java.lang.ClassCastException: org.apache.commons.fileupload.disk.DiskFileItem cannot be cast to [Ljava.lang.String;
org.apache.commons.fileupload.disk.DiskFileItem cannot be cast to [Ljava.lang.String;</ows:ExceptionText>
</ows:Exception>
</ows:ExceptionReport>
Presumably that isn't what you were looking for (though it isn't the full GetCap documnet)
Haha, not quite!
This works though (content sent via POST as data rather than as a form):
$ curl -d "SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&typeNames=WHSE_ENVIRONMENTAL_MONITORING.EMS_MONITORING_LOCN_GROUPS_SVW" https://delivery.openmaps.gov.bc.ca/geo/pub/wfs
<?xml version="1.0" encoding="UTF-8"?><xsd:schema xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:pub="http://delivery.openmaps.gov.bc.ca/geo/" xmlns:wfs="http://www.opengis.net/wfs/2.0" xmlns:xsd="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" targetNamespace="http://delivery.openmaps.gov.bc.ca/geo/">
<xsd:import namespace="http://www.opengis.net/gml/3.2" schemaLocation="http://delivery.openmaps.gov.bc.ca/geo/schemas/gml/3.2.1/gml.xsd"/>
<xsd:complexType name="WHSE_ENVIRONMENTAL_MONITORING.EMS_MONITORING_LOCN_GROUPS_SVWType">
<xsd:complexContent>
<xsd:extension base="gml:AbstractFeatureType">
<xsd:sequence>
<xsd:element maxOccurs="1" minOccurs="1" name="MONITORING_GROUP_ID" nillable="false" type="xsd:decimal"/>
<xsd:element maxOccurs="1" minOccurs="0" name="DESCRIPTION" nillable="true" type="xsd:string"/>
<xsd:element maxOccurs="1" minOccurs="1" name="GROUP_NAME" nillable="false" type="xsd:string"/>
<xsd:element maxOccurs="1" minOccurs="1" name="OFFICE_CD" nillable="false" type="xsd:string"/>
<xsd:element maxOccurs="1" minOccurs="1" name="AEMS_TAG" nillable="false" type="xsd:string"/>
<xsd:element maxOccurs="1" minOccurs="1" name="FEATURE_CODE" nillable="false" type="xsd:string"/>
<xsd:element maxOccurs="1" minOccurs="0" name="GEOMETRY" nillable="true" type="gml:GeometryPropertyType"/>
<xsd:element maxOccurs="1" minOccurs="0" name="OBJECTID" nillable="true" type="xsd:decimal"/>
<xsd:element maxOccurs="1" minOccurs="0" name="SE_ANNO_CAD_DATA" nillable="true" type="xsd:hexBinary"/>
</xsd:sequence>
</xsd:extension>
</xsd:complexContent>
</xsd:complexType>
<xsd:element name="WHSE_ENVIRONMENTAL_MONITORING.EMS_MONITORING_LOCN_GROUPS_SVW" substitutionGroup="gml:AbstractFeature" type="pub:WHSE_ENVIRONMENTAL_MONITORING.EMS_MONITORING_LOCN_GROUPS_SVWType"/>
</xsd:schema>
Let me test in the package
@ateucher yep that one is better. It would be good to know what the package is doing. In any case, I have rolled back the caching change in production so you can carry on with your work and we can continue testing.
Thanks for bringing this to my attention
Thanks @rustyjux all passing now! Really appreciate the quick attention :)
The http client we are using in the package sends the query as data via POST
, but I don't know the internal workings of it.
In general, I have tried to document how are using DSS web services here: https://github.com/bcgov/bcdata/blob/main/vignettes/service_documentation.Rmd
@rustyjux and @ateucher thank you!
No problem, sorry for the trouble. I will do some independent testing with that last POST request. If I follow up here can folks then try some testing in R?
I did a bit more reading on POST
- it looks like that style* is when you're actually sending data to fill a web form, so I'm not sure it's applicable here anyway?
*By "that style" I mean curl -X POST -F "param1=val1" -F "param2-val2" url
@rustyjux Anytime you want or need testing just open an issue and we will test.
thanks @stephhazlitt
@ateucher seems unlikely the package is using -F
for multipart/form-data
Hoping this provides a good proxy for the app:
curl -d "SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&typeNames=WHSE_ENVIRONMENTAL_MONITORING.EMS_MONITORING_LOCN_GROUPS_SVW" https://delivery.openmaps.gov.bc.ca/geo/pub/wfs
That for sure gets the basic case.. I can try to create a curl command than also involves sending data for a WFS query... might be a week or two before I get to it though
Ah it was an easy fix and had to do specifically with rewrites on requests to /geo/pub/wfs with no query and the need for an exception for POST there. Which is what I suspected and noted in my first comment... but I was on a call so didn't look closely at that piece. So much for multi-tasking.
If you can confirm using the package that all is well on delivery.openmaps.gov.bc.ca, I will re-implement caching in production with the added exception.
All is well from my perspective, thanks!
If you can confirm using the package that all is well on delivery.openmaps.gov.bc.ca, I will re-implement caching in production with the added exception.
All tests pass using delivery.openmaps.gov.bc.ca
And we're live in production. Again, my apologies. On the bright side, 5xx errors (mostly due to timeouts) on requests for the full GetCapabilities doc have dropped from ~15% of requests to 0.25%
No worries, thanks @rustyjux! I think it's smart to cache that document, and that's great to see that many fewer errors. We cache it locally within a session already, because we query it a lot. I guess not a lot of users are using POST
Looks like the service is down as Simon's bcdata fails too (ex. bcdata dump bc-airports > bc-airports.geojson
).
Hoping it is just a temporary glitch but posting here now anyway.
Interestingly - about an hour ago bcdc_get_data
was returning objects but bcdc_query_geodata
(vignette example) was not.
bcdata::bcdc_get_data("WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW")
#> Error: There was an issue sending this WFS request
#> ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
#> Request:
#> URL: https://openmaps.gov.bc.ca/geo/pub/wfs
#> POST fields:
#> SERVICE=WFS&VERSION=2.0.0&REQUEST=DescribeFeatureType&typeNames=WHSE_IMAGERY_AND_BASE_MAPS.GSR_AIRPORTS_SVW
#> Content-Type: application/x-www-form-urlencoded
#> Accept-Encoding: gzip, deflate
#> Accept: application/json, text/xml, application/xml, */*
#> User-Agent: https://github.com/bcgov/bcdata
#> Response:
#> status: HTTP/1.1 400
#> date: Thu, 04 Jul 2024 16:15:39 GMT
#> server: Apache
#> x-frame-options: allow-from (null)
#> x-control-flow-delay-ms: 0
#> content-type: application/xml
#> set-cookie: GS_FLOW_CONTROL=GS_CFLOW_-43486bae:1907e5f61f3:-7ff5
#> access-control-allow-origin: (null)
#> access-control-allow-credentials: true
#> access-control-allow-methods: POST, GET, OPTIONS, HEAD
#> access-control-allow-headers: X-Requested-With, Referer, Origin, Content-Type, SOAPAction, Authorization, Accept
#> access-control-max-age: 1000
#> connection: close
#> transfer-encoding: chunked
Created on 2024-07-04 with reprex v2.1.0
Services were down this morning. Should be back up. In future you can consult the Data Systems and Services Uptime page (look in MAP group) or more specifically the WMS and WFS checks.
Oddly the WFS checks didn't show an outage but WMS did:
Thank you for the response. hmm. new error with uptime showing that we are good to go.
Tried system restart and fresh cran download. Looks like Simon's bcdata is up and running again. Seems odd but I am good at breaking my machines so might just be me. Any chance you could confirm that you are all good? Looks like bcdc_get_record
runs but bcdata::bcdc_query_geodata
and bcdata::bcdc_get_data
choke
bcdata::bcdc_get_record("78ec5279-4534-49a1-97e8-9d315936f08b")
#> B.C. Data Catalogue Record: School Districts of BC
#> Name: school-districts-of-bc (ID: 78ec5279-4534-49a1-97e8-9d315936f08b)
#> Permalink:
#> https://catalogue.data.gov.bc.ca/dataset/78ec5279-4534-49a1-97e8-9d315936f08b
#> Licence: Open Government Licence - British Columbia
#> Description: The School Districts dataset contains the spatial
#> representation (polygon) of the current extent of the administrative
#> areas defined under section 176(1) of the School Act for the purposes
#> of preservation and promotion of the fundamental principle of local
#> autonomy and control of public education at the public and
#> governmental levels through locally elected school boards.
#> Available Resources (1):
#> 1. WMS getCapabilities request (wms)
#> Access the full 'Resources' data frame using:
#> bcdc_tidy_resources('78ec5279-4534-49a1-97e8-9d315936f08b')
#> Query and filter this data using:
#> bcdc_query_geodata('78ec5279-4534-49a1-97e8-9d315936f08b')
bcdata::bcdc_query_geodata("78ec5279-4534-49a1-97e8-9d315936f08b") |>
dplyr::filter(SCHOOL_DISTRICT_NAME %in% c("Greater Victoria", "Prince George","Kamloops/Thompson")) |>
dplyr::select(SCHOOL_DISTRICT_NAME) |>
dplyr::collect()
#> Error in `path_to_connection()`:
#> ! '' does not exist in current working directory
#> ('/private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/RtmpdnWOXx/reprex-6b74d586486-tweed-barb').
bcdata::bcdc_get_data("78ec5279-4534-49a1-97e8-9d315936f08b")
#> Error in `path_to_connection()`:
#> ! '' does not exist in current working directory
#> ('/private/var/folders/mg/h910y2c54fsc99qj74dyrjph0000gn/T/RtmpdnWOXx/reprex-6b74d586486-tweed-barb').
Created on 2024-07-04 with reprex v2.1.0
@NewGraphEnvironment I'm not on the Mapping Services team so I suggest you connect with them directly. I believe Jira is the preferred route - https://dpdd.atlassian.net/servicedesk/customer/portal/1/group/1/create/30
I can reproduce your error @NewGraphEnvironment - I'll look into it
Tracking the new issue in #339
I'm getting an error and several warnings when trying to access spatial data resources (I expect to get a spatial data frame).
This occurs when accessing spatial (but not-non spatial) data, via
bcdc_get_data()
andbcdc_query_geodata()
.Examples which create this error
Example which works fine:
bcdata::bcdc_get_data('d7e6c8c7-052f-4f06-b178-74c02c243ea4')
``` r library(bcdata) #> #> Attaching package: 'bcdata' #> The following object is masked from 'package:stats': #> #> filter # Problems bcdc_get_data(record = '76b1b7a3-2112-4444-857a-afccf7b20da8', resource = '4d0377d9-e8a1-429b-824f-0ce8f363512c') #> Warning in xml_find_all.xml_node(xml_res, "//xsd:sequence"): Undefined #> namespace prefix [1219] #> Warning: Unknown or uninitialised column: `type`. #> Warning: Unknown or uninitialised column: `nillable`. #> Error in `xml_df[, c("name", "nillable", "type")]`: #> ! Can't subset columns that don't exist. #> ✖ Columns `name` and `type` don't exist. #> Backtrace: #> ▆ #> 1. ├─bcdata::bcdc_get_data(...) #> 2. ├─bcdata:::bcdc_get_data.character(...) #> 3. │ ├─bcdata::bcdc_get_data(x, resource, verbose = verbose, ...) #> 4. │ └─bcdata:::bcdc_get_data.bcdc_record(...) #> 5. │ ├─bcdata::bcdc_query_geodata(record = record_id, ...) #> 6. │ └─bcdata:::bcdc_query_geodata.character(record = record_id, ...) #> 7. │ ├─bcdata::bcdc_query_geodata(obj, crs) #> 8. │ └─bcdata:::bcdc_query_geodata.bcdc_record(obj, crs) #> 9. │ └─bcdata:::feature_helper(query_list$typeNames) #> 10. │ ├─xml_df[, c("name", "nillable", "type")] #> 11. │ └─tibble:::`[.tbl_df`(xml_df, , c("name", "nillable", "type")) #> 12. │ └─tibble:::vectbl_as_col_location(...) #> 13. │ ├─tibble:::subclass_col_index_errors(...) #> 14. │ │ └─base::withCallingHandlers(...) #> 15. │ └─vctrs::vec_as_location(j, n, names, missing = "error", call = call) #> 16. └─vctrs (local) `Reprex