search

WikiWatershed / model-my-watershed

The web application front end for Model My Watershed.
https://modelmywatershed.org
Apache License 2.0
57 stars 31 forks source link

CINERGI searches failing #2867

Closed ajrobbins closed 6 years ago

ajrobbins commented 6 years ago

I'm observing 400 errors when attempting CINERGI searches on AoIs of various sizes:

screen shot 2018-06-04 at 2 58 33 pm

rajadain commented 6 years ago

Looks like CINERGI changed their API and didn't tell us 😞

Old Response ```json { "took": 141, "timed_out": false, "_shards": { "total": 5, "successful": 5, "failed": 0 }, "hits": { "total": 5779, "max_score": 4.464109, "hits": [ { "_index": "metadata_v5", "_type": "item", "_id": "4146e0ebc9a441988246314918a0869e", "_score": 4.464109, "_source": { "src_source_type_s": "WAF", "src_source_uri_s": "WAF:http://132.249.238.151:8080/cinergi/waf/U.S._Geoscience_Information_Network/", "src_source_name_s": "US GIN", "src_uri_s": "http://132.249.238.151:8080/cinergi/waf/U.S._Geoscience_Information_Network/850932de-b2b0-40a2-aa87-7645b23b5743.xml", "src_lastupdate_dt": "2017-01-14T15:23:24Z", "sys_created_dt": "2017-03-14T16:57:59.957Z", "sys_modified_dt": "2017-03-15T21:12:23.776Z", "sys_xmlmodified_dt": "2017-03-14T16:57:59.957Z", "sys_owner_s": "gptadmin", "sys_metadatatype_s": "iso19115-2", "fileid": "850932de-b2b0-40a2-aa87-7645b23b5743", "title": "US Water Bodies", "description": "U.S. Map Data Water Boundaries represents water feature areas within United States. Water boundaries include the following: basic hydrography, naturally flowing water features, man-made channels to transport water, inland bodies of water, man-made bodies of water, seaward bodies of water, bodies of water in a man-made excavation, and special water features.", "keywords_s": [ "Hydrography", "polygon", "water", "hydrography", "naturally flowing", "channel", "inland body", "seaward body", "man-made", "inlandWaters", "oceans", "transportation", "United States", "Channel", "BASIC", "Ocean", "Wetlands Reserve Program", "Flow", "Water", "geoscientificInformation" ], "links_s": "http://www.esri.com/", "contact_organizations_s": [ "Wetlands Reserve Program", "Wetlands Reserve Program" ], "contact_people_s": "Environmental Systems Research Institute", "apiso_Identifier_s": "850932de-b2b0-40a2-aa87-7645b23b5743", "apiso_Title_txt": "US Water Bodies", "apiso_Abstract_txt": "U.S. Map Data Water Boundaries represents water feature areas within United States. Water boundaries include the following: basic hydrography, naturally flowing water features, man-made channels to transport water, inland bodies of water, man-made bodies of water, seaward bodies of water, bodies of water in a man-made excavation, and special water features.", "apiso_OrganizationName_txt": "Wetlands Reserve Program", "apiso_Subject_txt": [ "Hydrography", "polygon", "water", "hydrography", "naturally flowing", "channel", "inland body", "seaward body", "man-made", "inlandWaters", "oceans", "transportation", "United States", "Channel", "BASIC", "Ocean", "Wetlands Reserve Program", "Flow", "Water", "geoscientificInformation" ], "apiso_TopicCategory_s": "geoscientificInformation", "apiso_KeywordType_s": [ "theme", "place", "theme", "theme", "theme", "theme", "theme", "theme" ], "apiso_Type_s": "dataset", "apiso_Modified_dt": "2014-10-30T11:58:56Z", "apiso_PublicationDate_dt": "1900-01-01T00:00:00Z", "apiso_HasSecurityConstraints_b": false, "envelope_geo": { "type": "envelope", "coordinates": [ [ -172.4378, 74.3889 ], [ -66.9497, 18.9108 ] ] }, "envelope_cen_pt": { "lat": 46.64985, "lon": -119.69375000000001 }, "timeperiod_nst": { "begin_dt": "1900-01-01T00:00:00Z", "begin_indeterminate_s": null, "end_dt": "1900-01-01T00:00:00Z", "end_indeterminate_s": null }, "apiso_Lineage_txt": "missing", "categories_cat": [ "Feature > Hydrologic Feature > Channel", "Organization > NON-GOVERNMENT/NON-PROFIT ORGANIZATIONS > BASIC", "Feature > Marine Feature > Ocean", "Property > Measure > Flow", "Material > Environmental Material > Water" ], "hierarchies_cat": [ "Category > Material > Environmental Material > Water", "Category > Property > Measure > Physical Quantity > Flow", "Category > Feature > Marine Feature > Marine Water Body > Ocean", "Category > Organization > NON-GOVERNMENT/NON-PROFIT ORGANIZATIONS > Bay Area Shared Information Consortium", "Category > Feature > Hydrologic Feature > Water Body > Watercourse > Channel" ] } }, { "_index": "metadata_v5", "_type": "item", "_id": "b1ca861ce6634ff09ee9ae7c0efd359b", "_score": 4.4625773, "_source": { "src_source_type_s": "WAF", "src_source_uri_s": "WAF:http://132.249.238.151:8080/cinergi/waf/ScienceBase_WAF/", "src_source_name_s": "ScienceBase_processed", "src_uri_s": "http://132.249.238.151:8080/cinergi/waf/ScienceBase_WAF/0080/4f4e4ae2e4b07f02db688d15.xml", "src_lastupdate_dt": "2017-01-15T22:03:15Z", "sys_created_dt": "2017-03-17T20:31:21.005Z", "sys_modified_dt": "2017-03-17T20:31:21.005Z", "sys_xmlmodified_dt": "2017-03-17T20:31:21.005Z", "sys_owner_s": "gptadmin", "sys_metadatatype_s": "iso19115-2", "fileid": "4f4e4ae2e4b07f02db688d15", "title": "Water resources data, Virginia, water year 2004 volume 2. Ground-water-level and ground-water-quality records", "description": "Water-resources data for the 2004 water year for Virginia consist of records of water levels and water quality of ground-water wells. This report (Volume 2. Ground-Water-Level and Ground-Water-Quality Records) contains water levels at 346 observation wells and water quality at 40 wells. Locations of these wells are shown on figures 4 through 9. The data in this report represent that part of the National Water Data System collected by the U.S. Geological Survey and cooperating State and Federal agencies in Virginia.", "keywords_s": [ "Water Data Report", "Observation", "Survey", "Well", "Ground Water Level", "Water Quality", "Volume", "Water", "Ground Water", "geoscientificInformation" ], "links_s": [ "https://www.sciencebase.gov/catalog/item/4f4e4ae2e4b07f02db688d15", "http://pubs.usgs.gov/wdr/2004/WDR-VA-04-2/", "http://pubs.er.usgs.gov/publication/wdrVA042", "https://www.sciencebase.gov/catalog/file/get/4f4e4ae2e4b07f02db688d15" ], "contact_organizations_s": [ "White, Roger K.", "Powell, Eugene D.", "Guyer, Joel R.", "Owens, Joseph A." ], "apiso_Identifier_s": "4f4e4ae2e4b07f02db688d15", "apiso_Title_txt": "Water resources data, Virginia, water year 2004 volume 2. Ground-water-level and ground-water-quality records", "apiso_Abstract_txt": "Water-resources data for the 2004 water year for Virginia consist of records of water levels and water quality of ground-water wells. This report (Volume 2. Ground-Water-Level and Ground-Water-Quality Records) contains water levels at 346 observation wells and water quality at 40 wells. Locations of these wells are shown on figures 4 through 9. The data in this report represent that part of the National Water Data System collected by the U.S. Geological Survey and cooperating State and Federal agencies in Virginia.", "apiso_OrganizationName_txt": "White, Roger K.", "apiso_Subject_txt": [ "Water Data Report", "Observation", "Survey", "Well", "Ground Water Level", "Water Quality", "Volume", "Water", "Ground Water", "geoscientificInformation" ], "apiso_TopicCategory_s": "geoscientificInformation", "apiso_KeywordType_s": [ "Publication", "theme", "theme", "theme", "theme", "theme" ], "apiso_Type_s": "dataset", "apiso_Modified_dt": "2014-07-21T20:48:37Z", "apiso_Language_s": "eng: USA", "apiso_HasSecurityConstraints_b": false, "envelope_geo": [ { "type": "envelope", "coordinates": [ [ -80.285748, 40.387116 ], [ -74.607748, 34.709116 ] ] }, { "type": "envelope", "coordinates": [ [ 29.932349, 31.222538 ], [ 29.942349, 31.212538 ] ] } ], "envelope_cen_pt": [ { "lat": 37.548116, "lon": -77.446748 }, { "lat": 31.217537999999998, "lon": 29.937348999999998 } ], "categories_cat": [ "Activity > Observation > Observation", "Activity > Observation > Survey", "Feature > Hydrologic Feature > Well", "Feature > Hydrologic Feature > Ground Water Level", "Property > Property (Other) > Water Quality", "Property > Measure > Volume", "Material > Environmental Material > Water", "Material > Environmental Material > Ground Water" ], "hierarchies_cat": [ "Category > Material > Environmental Material > Water", "Category > Material > Environmental Material > Water > Ground Water", "Category > Property > Measure > Volume", "Category > Property (Other) > Water Quality", "Category > Feature > Hydrologic Feature > Well", "Category > Feature > Hydrologic Feature > Ground Water Level", "Category > Activity > Observation", "Category > Activity > Observation > Survey" ] } } ] } } ```
New Response ```json { "start": 1, "num": 10, "total": 5847, "nextStart": 11, "sourceType": "Geoportal", "sourceKey": null, "results": [ { "id": "9acc3606f0f640d8bb49d0f30b10c39b", "title": "Water withdrawal and use in Maryland, 1990-91", "description": "REQUIRED FIELD", "published": "2017-03-17T19:51:06.352Z", "updated": "2017-03-17T19:51:06.352Z", "author": { "name": "gptadmin" }, "links": [ { "rel": "alternate", "type": "application/json", "href": "http://132.249.238.169:8080/geoportal/rest/metadata/item/9acc3606f0f640d8bb49d0f30b10c39b" }, { "rel": "alternate", "type": "text/html", "href": "http://132.249.238.169:8080/geoportal/rest/metadata/item/9acc3606f0f640d8bb49d0f30b10c39b/html" }, { "rel": "alternate", "type": "application/xml", "href": "http://132.249.238.169:8080/geoportal/rest/metadata/item/9acc3606f0f640d8bb49d0f30b10c39b/xml" } ], "categories": [ { "scheme": "keywords", "term": "USA, Maryland" }, { "scheme": "keywords", "term": "water use" }, { "scheme": "keywords", "term": "surface water" }, { "scheme": "keywords", "term": "groundwater" }, { "scheme": "keywords", "term": "water resources" }, { "scheme": "keywords", "term": "water demand" }, { "scheme": "keywords", "term": "water supply" }, { "scheme": "keywords", "term": "water resources development" }, { "scheme": "keywords", "term": "SW 4040 Water demand" }, { "scheme": "keywords", "term": "Water Resources Abstracts" }, { "scheme": "keywords", "term": "Water" }, { "scheme": "keywords", "term": "Groundwater" }, { "scheme": "keywords", "term": "Surface Water" }, { "scheme": "keywords", "term": "geoscientificInformation" } ], "point": { "x": -76.749967, "y": 39.000389 }, "bbox": { "xmin": -78.311967, "ymin": 37.438389, "xmax": -75.187967, "ymax": 40.562389 }, "_source": { "envelope_geo": { "coordinates": [ [ -78.311967, 40.562389 ], [ -75.187967, 37.438389 ] ], "type": "envelope" }, "apiso_TopicCategory_s": "geoscientificInformation", "src_lastupdate_dt": "2017-01-15T12:14:43Z", "apiso_Identifier_s": "5053881be4b097cd4fce204a", "description": "REQUIRED FIELD", "src_uri_s": "http://132.249.238.151:8080/cinergi/waf/ScienceBase_WAF/0060/5053881be4b097cd4fce204a.xml", "apiso_HasSecurityConstraints_b": false, "src_source_name_s": "ScienceBase (P)", "title": "Water withdrawal and use in Maryland, 1990-91", "hierarchies_cat": [ "Category > Material > Environmental Material > Water", "Category > Material > Environmental Material > Water > Ground Water", "Category > Material > Environmental Material > Water > Surface Water" ], "sys_owner_s": "gptadmin", "categories_cat": [ "Material > Environmental Material > Water", "Material > Environmental Material > Groundwater", "Material > Environmental Material > Surface Water" ], "keywords_s": [ "USA, Maryland", "water use", "surface water", "groundwater", "water resources", "water demand", "water supply", "water resources development", "SW 4040 Water demand", "Water Resources Abstracts", "Water", "Groundwater", "Surface Water", "geoscientificInformation" ], "apiso_Language_s": "eng: USA", "apiso_KeywordType_s": "theme", "fileid": "5053881be4b097cd4fce204a", "src_source_type_s": "WAF", "links_s": [ "https://www.sciencebase.gov/catalog/item/5053881be4b097cd4fce204a", "https://www.sciencebase.gov/catalog/file/get/5053881be4b097cd4fce204a" ], "sys_created_dt": "2017-03-17T19:51:06.352Z", "contact_organizations_s": [ "JC Wheeler", "US GEOLOGICAL SURVEY, EARTH SCIENCE INFORMATION CENTER, OPEN-FILE REPORTS SECTION, BOX 25286, MS 517, DENVER, CO 80225 (USA)" ], "apiso_Title_txt": "Water withdrawal and use in Maryland, 1990-91", "sys_xmlmodified_dt": "2017-03-17T19:51:06.352Z", "sys_metadatatype_s": "iso19115-2", "apiso_OrganizationName_txt": "JC Wheeler", "apiso_Subject_txt": [ "USA, Maryland", "water use", "surface water", "groundwater", "water resources", "water demand", "water supply", "water resources development", "SW 4040 Water demand", "Water Resources Abstracts", "Water", "Groundwater", "Surface Water", "geoscientificInformation" ], "envelope_cen_pt": { "lon": -76.749967, "lat": 39.000389 }, "apiso_Abstract_txt": "REQUIRED FIELD", "apiso_Modified_dt": "2014-06-26T10:58:05Z", "apiso_Type_s": "dataset", "sys_modified_dt": "2017-03-17T19:51:06.352Z", "src_source_uri_s": "WAF:http://132.249.238.151:8080/cinergi/waf/ScienceBase_WAF/" } }, { "id": "bd46f5325916481ea722e90c5d9ccf06", "title": "Water Resources Data, Virginia, Water Year 1990. Volume 2: Ground-Water Level and Ground-Water Quality Records", "description": "REQUIRED FIELD", "published": "2017-03-17T18:43:07.237Z", "updated": "2017-03-17T18:43:07.237Z", "author": { "name": "gptadmin" }, "links": [ { "rel": "alternate", "type": "application/json", "href": "http://132.249.238.169:8080/geoportal/rest/metadata/item/bd46f5325916481ea722e90c5d9ccf06" }, { "rel": "alternate", "type": "text/html", "href": "http://132.249.238.169:8080/geoportal/rest/metadata/item/bd46f5325916481ea722e90c5d9ccf06/html" }, { "rel": "alternate", "type": "application/xml", "href": "http://132.249.238.169:8080/geoportal/rest/metadata/item/bd46f5325916481ea722e90c5d9ccf06/xml" } ], "categories": [ { "scheme": "keywords", "term": "Water Data Report" }, { "scheme": "keywords", "term": "Ground Water Level" }, { "scheme": "keywords", "term": "Volume" }, { "scheme": "keywords", "term": "Water" }, { "scheme": "keywords", "term": "Ground Water" }, { "scheme": "keywords", "term": "geoscientificInformation" } ], "point": { "x": -77.446748, "y": 37.548116 }, "bbox": { "xmin": -80.285748, "ymin": 34.709116, "xmax": -74.607748, "ymax": 40.387116 }, "_source": { "envelope_geo": { "coordinates": [ [ -80.285748, 40.387116 ], [ -74.607748, 34.709116 ] ], "type": "envelope" }, "apiso_TopicCategory_s": "geoscientificInformation", "src_lastupdate_dt": "2017-01-14T20:33:56Z", "apiso_Identifier_s": "4f4e4a09e4b07f02db5fa819", "description": "REQUIRED FIELD", "src_uri_s": "http://132.249.238.151:8080/cinergi/waf/ScienceBase_WAF/0030/4f4e4a09e4b07f02db5fa819.xml", "apiso_HasSecurityConstraints_b": false, "src_source_name_s": "ScienceBase (P)", "title": "Water Resources Data, Virginia, Water Year 1990. Volume 2: Ground-Water Level and Ground-Water Quality Records", "hierarchies_cat": [ "Category > Material > Environmental Material > Water", "Category > Material > Environmental Material > Water > Ground Water", "Category > Property > Measure > Volume", "Category > Feature > Hydrologic Feature > Ground Water Level" ], "sys_owner_s": "gptadmin", "categories_cat": [ "Feature > Hydrologic Feature > Ground Water Level", "Property > Measure > Volume", "Material > Environmental Material > Water", "Material > Environmental Material > Ground Water" ], "keywords_s": [ "Water Data Report", "Ground Water Level", "Volume", "Water", "Ground Water", "geoscientificInformation" ], "apiso_Language_s": "eng: USA", "apiso_KeywordType_s": [ "Publication", "theme", "theme", "theme" ], "fileid": "4f4e4a09e4b07f02db5fa819", "src_source_type_s": "WAF", "links_s": [ "https://www.sciencebase.gov/catalog/item/4f4e4a09e4b07f02db5fa819", "http://pubs.er.usgs.gov/publication/wdrVA902", "https://www.sciencebase.gov/catalog/file/get/4f4e4a09e4b07f02db5fa819" ], "sys_created_dt": "2017-03-17T18:43:07.237Z", "contact_organizations_s": [ "Prugh, B. J.", "Easton, F. J." ], "apiso_Title_txt": "Water Resources Data, Virginia, Water Year 1990. Volume 2: Ground-Water Level and Ground-Water Quality Records", "sys_xmlmodified_dt": "2017-03-17T18:43:07.237Z", "sys_metadatatype_s": "iso19115-2", "apiso_OrganizationName_txt": "Prugh, B. J.", "apiso_Subject_txt": [ "Water Data Report", "Ground Water Level", "Volume", "Water", "Ground Water", "geoscientificInformation" ], "envelope_cen_pt": { "lon": -77.446748, "lat": 37.548116 }, "apiso_Abstract_txt": "REQUIRED FIELD", "apiso_Modified_dt": "2014-07-21T20:45:21Z", "apiso_Type_s": "dataset", "sys_modified_dt": "2017-03-17T18:43:07.237Z", "src_source_uri_s": "WAF:http://132.249.238.151:8080/cinergi/waf/ScienceBase_WAF/" } } ] } ```

I think we can use the _source field of the new response, so should be doable, but it would be nice in the future to know about breaking API changes before they are pushed to production.

ajrobbins commented 6 years ago

@aufdenkampe Tagging you here so you're aware of this...we will adjust the way we consume the CINERGI API so it works again in the app, but down the road we may run into trouble if we are off-contract and Hydroshare/CUAHSI/CINERGI make breaking API changes.

aufdenkampe commented 6 years ago

@izaslavsky, @valentinedwv and @tom2275

The Model My Watershed / WikiWatershed app has been happily searching CINERGI resources for nearly 9 months, but that just broke! Happily, @rajadain figured out the change in the API and fixed it, and we are about to make a new release, which won't leave our 16,000 users hanging for long.

Could you let us know next time you update your API, so that we can make accommodations in advance? Thanks!

cc: @emiliom

valentinedwv commented 6 years ago

Apologize. We updated the codebase to Geoportal 2.6. Did not know that the opesearch results changed.