search

GIScience / ohsome-quality-api

Data quality estimations for OpenStreetMap
https://api.quality.ohsome.org
GNU Affero General Public License v3.0
35 stars 7 forks source link

API refuses (previously working) ohsome responses as input. #714

Closed kolumdium closed 1 year ago

kolumdium commented 1 year ago

I was able to fetch results from the ohsome api. 

def get_ohsome_result(url="https://api.ohsome.org/v1/elements/length", name="Freiberg, Sachsen", filter_string=None):

    bbox = ox.geocoder.geocode_to_gdf(name, which_result=0)
    north = bbox.bbox_north[0]
    east = bbox.bbox_east[0]
    south = bbox.bbox_south[0]
    west = bbox.bbox_west[0]

    parameters = {
        "bboxes": "{},{},{},{}".format(west, south, east, north),
        "filter": "{}".format(filter_string),
        "format": "json",
        "time": "2008-01-01/2023-01-01/P1M",
    }
    headers = {
        "accept": "application/json",
        "Content-Type": "application/x-www-form-urlencoded",
    }

    response = requests.post(url, data=parameters, headers=headers)
    response.raise_for_status()  # Raise an Exception if HTTP Status Code is not 200

    ohsome_result = response.json()["result"]
    return ohsome_result

And Input them into the oqt API as input.

def get_oqt_result(url="https://oqt.ohsome.org/api/indicators/mapping-saturation", bpolys=None, ohsome_result=None, filter_string=None):

    topic = {
        "key": "mapping-saturation-cycleways",
        "name": "Cycleways",
        "description": "Number of features taged as {}".format(filter_string),
        "data": {"result": ohsome_result}
    }

    parameters = {
        "bpolys": bpolys,
        "topic": topic,
        "includeSvg": True,
        "includeHtml": False,
        "flatten": False
    }

    headers = {"Content-Type": "application/json",
               "Accept": "application/json"}
    response = requests.post(url, json=parameters, headers=headers)
    response.raise_for_status()  # Raise an Exception if HTTP Status Code is not 200

    oqt_result = response.json()
    return oqt_result

Such a request gets denied now with RequestValidationError. Even after adjusting to the new bpoly format a previously working request gets denied.

Full Response log. 

{
  "apiVersion": "1.0.1",
  "type": "RequestValidationError",
  "detail": [
    {
      "type": "enum",
      "loc": [
        "body",
        "topic"
      ],
      "msg": "Input should be 'building-count','building-area','major-roads-length','railway-length','poi','schools','kindergarten','clinics','doctors','bus-stops','tram-stops','subway-stations','supermarkets','marketplaces','parks','forests','fitness-centres','fire-stations','hospitals','amenities','landmarks','mapaction-settlements-count','mapaction-capital-city-count','mapaction-rail-length','mapaction-major-roads-length','mapaction-lakes-count','mapaction-lakes-area','mapaction-rivers-length','lulc','local-food-shops','fast-food-restaurants','restaurants','convenience-stores','pubs-and-biergartens','alcohol-and-beverages','sweets-and-pasteries','clc-arable-land-area','clc-permanent-crops-area','clc-pastures-area','clc-forest-area','clc-leaf-type','clc-shrub-area','clc-open-spaces-area','clc-wetland-area','clc-water-area','clc-waterway-len','minimal' or 'infrastructure-lines'",
      "input": {
        "key": "mapping-saturation",
        "name": "Cycleways",
        "description": "Number of features taged as highway=* and geometry:line",
        "data": {
          "result": [
            {
              "timestamp": "2008-01-01T00:00:00Z",
              "value": 87197.26
            },
            {
              "timestamp": "2008-02-01T00:00:00Z",
              "value": 93851.79
            },
            {
              "timestamp": "2008-03-01T00:00:00Z",
              "value": 93345.01
            },
           [...]
            {
              "timestamp": "2022-10-01T00:00:00Z",
              "value": 857658.21
            },
            {
              "timestamp": "2022-11-01T00:00:00Z",
              "value": 858757.51
            },
            {
              "timestamp": "2022-12-01T00:00:00Z",
              "value": 859361.89
            },
            {
              "timestamp": "2023-01-01T00:00:00Z",
              "value": 860655.16
            }
          ]
        }
      },
      "ctx": {
        "expected": "'building-count','building-area','major-roads-length','railway-length','poi','schools','kindergarten','clinics','doctors','bus-stops','tram-stops','subway-stations','supermarkets','marketplaces','parks','forests','fitness-centres','fire-stations','hospitals','amenities','landmarks','mapaction-settlements-count','mapaction-capital-city-count','mapaction-rail-length','mapaction-major-roads-length','mapaction-lakes-count','mapaction-lakes-area','mapaction-rivers-length','lulc','local-food-shops','fast-food-restaurants','restaurants','convenience-stores','pubs-and-biergartens','alcohol-and-beverages','sweets-and-pasteries','clc-arable-land-area','clc-permanent-crops-area','clc-pastures-area','clc-forest-area','clc-leaf-type','clc-shrub-area','clc-open-spaces-area','clc-wetland-area','clc-water-area','clc-waterway-len','minimal' or 'infrastructure-lines'"
      }
    },
    {
      "type": "extra_forbidden",
      "loc": [
        "body",
        "includeSvg"
      ],
      "msg": "Extra inputs are not permitted",
      "input": true,
      "url": "https://errors.pydantic.dev/2.2/v/extra_forbidden"
    },
    {
      "type": "extra_forbidden",
      "loc": [
        "body",
        "includeHtml"
      ],
      "msg": "Extra inputs are not permitted",
      "input": false,
      "url": "https://errors.pydantic.dev/2.2/v/extra_forbidden"
    },
    {
      "type": "extra_forbidden",
      "loc": [
        "body",
        "flatten"
      ],
      "msg": "Extra inputs are not permitted",
      "input": false,
      "url": "https://errors.pydantic.dev/2.2/v/extra_forbidden"
    }
  ]
}

I am not sure if this is intentional or not. But I would like to continue using oqt with ohsome for arbitrary filters. I can't rule out a user error. But I could not find out the problem on my site. If it is an error on my site, I request clarification on using "input". I would also like to know if this is a feature no longer provided. Thank you very much.

Hagellach37 commented 1 year ago

Hey @kolumdium , thanks for reaching out here and opening the issue. Glad to see that you are already using the API.

The short answer to your question is that you need to use a slightly adjusted endpoint path now: https://api.quality.ohsome.org/v1/indicators/mapping-saturation/data.

I think @matthiasschaub might be able to provide us some more details.

matthiasschaub commented 1 year ago

Hello @kolumdium, -- @Hagellach37 is right: Please update your base URL and endpoint. Also, always send GeoJSON FeatureCollections (I think you already figured that out).

I hope that helps. Please let us know if you are able to get your code working.

I am curious, what is your cycleways filter query?

kolumdium commented 1 year ago

Hello @matthiasschaub and @Hagellach37,

Thank you for your assistance. I've successfully resolved the issue by changing the endpoint, as suggested.

API Documentation

It wasn't immediately clear where this endpoint change was documented. Could this information have been found somewhere other than opening a GitHub issue? The API documentation I referred to did not mention the new /data entrypoint. Is there a way to subscribe for notifications about API changes?

API Changes

I also noticed that the new endpoint no longer supports the arguments includeSvg, includeHtml, or flatten. Additionally, the resultskey in the response has been renamed to result. I'm noting this for the sake of completeness of this issue. Also does that mean I have to plot the svg myself now?

Cycleway Query

Regarding my cycleway query: As you probably know OSM offers numerous tagging options for cycleways, which can lead to verbose queries. Here's the filter I'm using currently:

"filter": "(highway=cycleway 
or cycleway in (lane,crossing,track,shared_lane,opposite,shared,opposite_lane,yes,share_busway,sidepath,opposite_track) 
or cycleway:left in (lane,crossing,track,shared_lane,opposite,shared,opposite_lane,yes,share_busway,sidepath,opposite_track) 
or cycleway:right in (lane,crossing,track,shared_lane,opposite,shared,opposite_lane,yes,share_busway,sidepath,opposite_track)
or bicycle in (yes,designated,use_sidepath,permissive,official,optional_sidepath) 
or living_street=yes) 
and geometry:line"

This filter seems to provide a comprehensive coverage of paths that are accessible to bicycles, as also discussed in Plank et al..

Looking forward to working with this API.

matthiasschaub commented 1 year ago

We document changes (especially breaking changes) of each release in our CHANGELOG. Usually we even include a "How to update" section in our changelog. Development happened quite fast over the last months, and we did a lot of breaking changes as you experienced yourself. But now we released a 1.0 version and do not plan to implement any breaking features in the near future.

In regard to the documentation: The endpoint you are using is not a feature we want to promote and document for now. The main reason is that it works for the Mapping Saturation indicator and not for the others. That said, we do not plan to discontinue this endpoint, it's just hidden from the docs for now.

include_svg and include_html are removed in favor of include_figure. We do not return an SVG anymore but a Plotly figure. For how to load and display this figure from the response in Python take a look at our examples: https://github.com/GIScience/ohsome-quality-api

flatten is also removed in favor of showcasing a workflow how to work with oqapi results in QGIS: https://heigit.org/visualizing-oqt-api-results-in-qgis/

Thanks for filling out this issue. If you still have questions, please ask!

Also thanks for sharing the filter query and research paper :) Will take a look at those.