search

iobis / obis-issues

Repository for all OBIS related issues and feature requests
5 stars 3 forks source link

API route for listing Nodes and Node stats #147

Open skybristol opened 4 years ago

skybristol commented 4 years ago

Looking at a spreadsheet that @wardappeltans put together for SG-OBIS-8, I wondered why we don't just have an API route that would return basic status on all OBIS Nodes, including an indication of when they last contributed data. I noticed that the current API does not have a route that will simply return basic information for all Nodes; you have to know a node ID before you can query it. We should have a route that simply returns all Nodes. Ideally, the response would include something like a time series on records per year or something that could be used to produce Ward's spreadsheet.

Also, it would be nice if the API route followed the pattern that GitHub and many other systems use, where all nodes and individual node responses explicitly list other routes in the API accepting node ID as an input parameter. That way, anyone can quickly write code to follow a Node and get occurrences, stats, and other responses with REST URLs pre-built into the response.

pieterprovoost commented 4 years ago

It seems that the node API documentation is incomplete. There is an endpoint to get all nodes, although the information you are after is not there (yet): https://api.obis.org/node

skybristol commented 4 years ago

Well, that's cool. I don't see any other API route at the moment that tells us when datasets or records were actually submitted/pulled from source. We could go to the IPTs in the node references for that, but I was hoping for something straight off the API that would provide the report, even if it required a followup after retrieving the nodes.

As an imperfect way of using the API, I added a couple of sheets to the spreadsheet that try to recreate Ward's list dynamically. The first uses a custom ImportJSON function to grab the node info and then the built in ImportXML function to hit the IPT for the first item (which should be the latest if following RSS spec) and get the pubDate. The date is parsed to get the year (since the string doesn't evaluate as a date without some tweaking). The last sheet uses the query function to grab the node name and latest year from the dynamic data to answer the question of when the last posted data.

There are some differences in the list generated dynamically from OBIS and Ward's list, so maybe I'm missing something. The OBIS API reports 32 nodes whereas the static list only has 23 (22 if I hadn't added OBIS USA). And the yes/no values don't quite line up. Did data for nodes get to the system some other way?

So, maybe this doesn't work quite yet, but we should have a way for OBIS to fully report on itself.

P.S. If we want people to use the system for building apps, we do need to fix the Swagger/OpenAPI interface still so that it actually responds inline to trying queries.