search

openreferral / specification

The Human Services Data Specification - a data exchange format developed by the Open Referral Initiative
https://openreferral.org
Other
117 stars 50 forks source link

Ideas for publishing records in HSDS #118

Closed NeilMcKLogic closed 4 years ago

NeilMcKLogic commented 7 years ago

As multiple of us consider how to architect an effective API for HSDS, I'd like to hear thoughts of others about how to emit records in HSDS, in particular, related records. Our collective preliminary work will help us all prepare for an eventual standard, critical for maximizing interoperability.

For example, with an API call, if someone requests a Service record, in addition to the objects directly related to it (phone, contact, eligibility, etc), should you also return to them its parent Organization (and its related objects)? What about also returning the Location records linked to the Service (via service_at_location) ? This seems sensible to me as you very likely will need all of that information related to the requested Service, but of course it increases the payload.

The same question applies for scenarios where one is requesting a Location object.

Perhaps the ultimate answer is to always return the full Organization record and all of its child entries (all Services and Locations and their related objects), even if the request was just for a particular Service or Location. You'll get "everything" for the Organization, which might really bloat the response but then you can work with the data locally.

Or maybe this just becomes an option in the API to indicate what you want returned in the response.

Any thoughts from other folks here on these various approaches?

timgdavies commented 7 years ago

@NeilMcKechnie We've just been talking today to @kinlane about plans to move forward formal API spec to go along with Open Referral. I'm sure this will feed into that.

I'll create a milestone for API work and associate this there, but hope @kinlane and others can jump in with further thoughts on the approach for this.

kinlane commented 7 years ago

Getting back to thoughts guys. I'd love to find a way forward for this. More specifically I want to "move forward formal spec to go along with Open Referral", but I also want to help move forward an API spec for each implementation that overlaps with it (iCarol, etc.).

I was working on this before checking out last summer and would like to help steward the process, and definition(s) forward. What this requires is us to speak a common language for this specification via a common platform where we can record changes, and resulting document.

I use OpenAPI Spec (openapis.org) as an open, machine readable format to facilitate this process. It is the leading standard for doing this, which is currently in Linux Foundation. OpenAPI Spec gives us a language to describe the Open Referral in JSON Schema, as well as the surface area of the API that governs responses & requests of the underlying schema.

I have an OpenAPI Spec from earlier work(http://adopta-agency.github.io/open-referral-api/), with the JSON version of the machine-readable definition (http://adopta-agency.github.io/open-referral-api/api-commons/openapi-spec.json), as well as a YAML edition (https://github.com/adopta-agency/open-referral-api/blob/gh-pages/api-commons/openapi-spec.yaml) -- as OpenAPI Spec allows for JSON or YAML editions, speaking to wider, less technical audience.

This is the common language that API providers are using to communicate the API design with other stakeholders, in a precise, and machine readable way. OpenAPI Spec can be used to define the master Open Referral definition, as well as the definition of each individual deployment or integration. This opens up the opportunity to conduct a "diff" between the definitions, showing what is compatible, and what is not.

The natural platform for facilitating this discuss is Github, as it provides the version control, diff, communication, and user interactions throughout the process. Allowing each path, parameter, response, request, and other elements to be discussed independently. I've also used & developed other tooling for annotating the API definition as part of the process (http://apis.json.annotation.apievangelist.com/).

This approach requires that all stakeholders involved become fluent in OpenAPI Spec, and JSON Schema, but is something that I've worked with technical, as well as non-technical folks to achieve. This process allows us all to be on the same page, with all discussion around the Open Referral API definition and schema being open and accessible to the public, with the ability to invite and include new participants at any time using Github's existing tooling.

Once a formal draft API specification + underlying JSON schema for Open Referral is established, it becomes the machine readable contract, and central source of truth regarding the API definition as well as the data model schema--one that humans can follow, as well as be used to drive almost every other stop along the API life cycle like deployment, mocking, management, testing, monitoring, SDKs, documentation, etc.

Let me know thoughts. Concerns. And I'm happy to kick off the formal process on any repositories that the stakeholder organizations dictate.

timgdavies commented 7 years ago

Thanks Kin. This is really useful. Thoughts below:

Managing the specification

So: right now we have the 'formal specification' of the HSDS in this datapackage.json file which describes how to serialise a set of data in flat files.

But - I think it should be fairly straightforward to either in version 1.1:

  1. Generate the JSON Schema definitions you need in an OpenAPI spec from this; or
  2. Switch to using a JSON Schema as the authoritative schema, and then generate datapackage.json from that;

The one issue I see with (2) is that Data Packages have a concept of foreign keys in a way I believe JSON schema doesn't fully handle.

API design issues

I think the idea that HSDS core specifies the flat tables of data, and then an associated standard API spec describes how to provide that to users interactively, without specifying a particular canonical JSON hierarchy of HSDS, is well worth exploring.

Back on @NeilMcKechnie's original question - the one thing I wasn't sure how to represent in Open API spec is the question of which related objects should be contained in a response?

Right now, HSDS doesn't have a set hierarchy (e.g. you might go in via 'Services' and expect to find their 'Locations', or you might go in via 'Locations' and expect to find the 'Services'), so it doesn't seem as simple as modelling a structure in the JSON schema itself (as I think we could end up with circulate references etc.)

I notice in the JSON-API Specs they talk about a pattern of having an ?include= parameter to specify which relationships should be expanded out in a result, and the option of APIs have a default set of compound objects they generally give. Is this a pattern we might want to explore? And something that the Open API spec could be used to easily describe?

Patching schemas

We've been exploring use of JSON Merge Patch (Python implementation) with the Open Contracting Data Standard as a good way of maintaining 'diff'ed' or 'extended' versions of a schema for particular use-cases.

User participation

I really like your annotation approach and GitHub integration. Well worth exploring.

NeilMcKLogic commented 7 years ago

Thanks Kin and Tim for chiming in. The immediacy of developing a standards-based approach has become strong in recent weeks. We have no less than four concurrent projects building HSDS-based APIs on our own platform and with at least one client building their own, as well as applications that are consuming those APIs.

Tim your comment about what related tables to include in a response is one of several major considerations to iron out. Another one is the verbs and RESTful structure we'll use.

Speaking as someone who is both actively developing these APIs and also building software to consume them, literally this week, myself and the iCarol engineers are undertaking these questions right now.

If interested parties are willing to collaborate with us now on these kinds of technical details (Kin? Tim?) we would much more likely end up with stronger alignment for an HSDS API standard. Either way, my team will continue forging forward based on our internal and client needs.

timgdavies commented 7 years ago

@NeilMcKechnie Are there any discussions / docs visible from the parallel work you have going on that we could us to make a quick comparison of areas of difference / distance to be able to prioritise parallel work.

NeilMcKLogic commented 7 years ago

We do but they are not (yet?) items I'd want to share semi-publicly here on Github. Should we connect directly by email perhaps?

greggish commented 7 years ago

I'm at a conference this week, so I'm just lightly tracking these threads. I do want to chime in briefly here: Open Referral has received funding for a formal development process along these very lines. I've been told that we will be able to announce details about this grant sometime this month, and have been tentatively planning to kick off the process in March. I can say that I expect to be working formally with Kin as our technical lead (in dialogue with Tim & co as they put together HSDS 1.1), and we'll also be working in partnership with various leadership from the information-and-referral field.

Sorry I can't share more details right this moment; I don't want to hold anyone back, so perhaps Kin and Neil should go ahead and start discussing. If folks feel like March is late for some reasons, please let me know and we'll figure something out. Overall I'm v excited for this opportunity and thrilled to see y'all already hashing out some of these points.

kinlane commented 7 years ago

@timgdavies:

datapackage.json (KL) file is fine for me to work from, you just keep working, and I'll parse. Lack of foreign keys is issue - (KL) Definitely something that is issue in translating to API world. Sometimes it is dealt with through API design, and doesn't need reflecting, other times I rears head as ugly challenge...good thing about JSON specs, is we can use x- to identify out of spec references until we find solution, or spec catches up. Request / Response structure - The one thing I wasn't sure how to represent in Open API spec is the question of which related objects should be contained in a response? - (KL) always a concern. always something that needs iteration in API design. There are several approaches to tailoring / personalizing response object in requests, and we will have to weigh them. JSON Merge Patch - (KL) Very suitable solution. Will look to use or emulate as part. Annotation Approach - (KL) Will keep working on to evolve with this project as well.

@NeilMcKechnie - Glad time is ripe, and your team is work on. Just keep on, output whatever you can as far s documentation to share.

@all - My goal is to help be one of the stewards of Open Referral API def & schema, and do so with many complimenting, contributing, and even competing projects. It should always guide, and influence individual projects, but they will always have their own constraints.

Let me pick up where I left off with the first version of the OpenAPI Def + JSON schema definitions for Open Referral. Let me take the datapackage.json you have for Miami and process, and contrast as well. @NeilMcKechnie if your team can provide any documentation regarding API definition and schema used, preferably as machine-readable as possible, but I can process anything. I will work to keep the 3 in sync, and stand up the annotation discussion and see what we can produce.

NeilMcKLogic commented 7 years ago

Thanks Kin, I'll get something pulled together to provide as a reference and of course will remain active with you and the other contributors as this project progresses.

kinlane commented 7 years ago

Hey Y'all

I just wanted to update where I am with my work. I am ready for a conversation around the HSDS spec, as well as the implementation known as Switchboard of Miami:

I will be maintaining an OpenAPI Spec, JSON Schema, and Data Package for each project with all things flowing up to the master HSDS spec -- currently being driven by realities of Miami Ohana implementation.

The current definition (for both) is verbose, autogenerated from the Data Package, not meant to draft. It is just meant to be the conversation starter and is something I will maintain. No extra work needed from y'all here.

I am happy to setup a Github repository to manage the HSDS to iCarol and facilitate the conversation there as well.

When possible I'd like to get technical details for the Miami project:

I'll be asking similar questions + any docs from iCarol when it makes sense, and you are ready.

Next, I am working on the following:

Don't worry about engaging with the sites I provided, I will keep in sync with any technical details I receive from each project. Then, as it makes sense I will bring you guys into specific Github issue conversation threads, about specific portions of the specification. Similar to what we are doing here.

--Thanks, Kin Lane.

JimSaiya commented 7 years ago

In response to something @kinlane mentioned a couple comments ago in this thread:

Lack of foreign keys is issue - ... good thing about JSON specs, is we can use x- to identify out of spec references until we find solution, or spec catches up.

That made me think of an IETF RFC you might want to consider: http://www.rfc-base.org/rfc-6648.html

Here's another GitHub project that wrestled with that issue at one point: https://github.com/angular/angular.js/issues/10903

Just FYI, Jim

timgdavies commented 7 years ago

Thanks @kinlane. This makes sense. I'll try and get the datapackage.json for 1.0 fully sorted in the next week, and then we'll be getting close to 1.1.

On that note - do you have a versioning approach for the sites? Might be good if we put a bit 'alpha' banner on the general HSDS spec one until we've got 1.1 approved and also, as part of that process, put the API spec design patterns to people.

Assuming the Switchboard site will follow Ohana, are you thinking for the HSDS API design patterns of just building on that with tweaks, or doing some rethinking of the model as well?

NeilMcKLogic commented 7 years ago

I've looked a bit at Ohana and it is rather different than the iCarol API. Not saying ours is better necessarily but there are very likely good ideas to be gleaned from both camps. In particular the parameters for filtering and sorting searches are pretty important to get right so that it is useful.

Also I believe Ohana is pretty "location-centric" meaning you can't get to services without first going through locations. That would be a pretty major "oops" in a generally useful standard because most searches we see look first for services and then sort the resulting locations by proximity.

There are likely more such a-ha's but until Kin is ready to get into the spec writing its probably not useful to identify and enumerate all of them just yet.

monfresh commented 7 years ago

I'm not sure what you mean by "can't get to services without first going through locations". In Ohana API, you can search for one or more "categories", which map to one of the terms in the Open Eligibility taxonomy. For example, if you wanted to search for services that match the terms "Transit" or "Care", you would do this: https://ohana-api-demo.herokuapp.com/api/search?category[]=Transit&category[]=Care

This will automatically sort those services by location in just one API call.

See the search documentation here: https://codeforamerica.github.io/ohana-api-docs/#get-locations-that-match-certain-criteria-

NeilMcKLogic commented 7 years ago

Hi Moncef, thanks for the correction. I was sort of quoting information I had heard third hand and should have been more diligent before writing/posting that. Sorry about that.

greggish commented 7 years ago

I might have been the one from whom Neil heard that third hand. But I heard that from several developers who implemented Ohana. In any case, we've learned a fair amount from our Miami implementation of Ohana and will compile it to share later this month or early in March -- in time for the API standardization project that Kin will lead this spring / summer.

kinlane commented 7 years ago

Getting back to this work, and what I've missed.

@JimSaiya definitely agree on x-....not ever the ideal. In this case I'd lean more heavily on API design to solve problems of relationships and have them expressed in URI over needing to go down and dirty with an x- in a specification.

@timgdavies:

Once I have a live base URL to hit I can start certifying the availability of each endpoint (reality) for Switchboard of Miami, then we can narrow down the OpenAPI Spec, and I can finish up documentation, code samples, and get the portal more ready for testing by others.

I'm also looking for any UI / Graphical guidance on portal -- as it's just a Bootstrap default right now --- Greg? What should it look like? I can do work...just need direction.

If we don't have any development base URL for the existing Switchboard Miami API, and we are in agreement that it is Ohana+, I can just get to work narrowing down based upon existing definition.

Hope that makes sense...thanks.

greggish commented 7 years ago

Thanks Kin.

re the portal UI, i think that's an open question. I don't have a clearly scoped feature set. I could take a look and make my own notes on what you currently have, and i can also bring notes from my interviews with potential API consumers. suggestions would be welcome for methodology on each count

~g

kinlane commented 7 years ago

@greggish -- at this point I say we leave it as Twitter Bootstrap. Deal with as part of usability testing with a functional API.

kinlane commented 7 years ago

I updated the OpenAPI Spec for the Switchboard Miami developer portal. I whittled it down to only have the paths and parameters from the API definition for Ohana Heroku demo app.

I am trying to focus on an API definition for what we have (Ohana), and Switchboard of Miami as the implementation -- working out the workflow on a handful of the core paths, available in current Ohana implementation:

/api/organizations

/api/organizations/{organization}

/api/organizations/{organization}/locations

/api/locations

/api/locations/{location}

/api/search

Hammer out the first draft of HSDS API definition for exactly what is accessible via Ohana implementation, providing GET access to these resources. Then we can talk about POST, PUT, and DELETE availability.

Then we can also discuss any additional paths, parameters that are available (I just don't know), as well as link up to the underlying schema. I can take a stab at these, but really I need a working API to hit, and see what responses are there -- sorry, flying blind here.

Then I want to do the same for any iCarol implementation I can get my hands on, which will then impact my suggestions for a true version 1.0 of the HSDS API definition which includes version 1.0 of the data specification.

Right now the master spec I have, which also drives the master API portal, is an auto-generated, comprehensive view of what the HSDS API specification should be, not a snapshot of what we currently have, and what iCarol brings to the table.

Again, happy to facilitate discussion further...just thinking out loud here.

robredpath commented 7 years ago

I think @greggish might have already passed this on, but there's a running instance of the existing API loaded with Miami data at ohana-api.herokuapp.com/api/

NeilMcKLogic commented 7 years ago

Hey @robredpath do you know if the Miami instance has been updated with the recent tweaks that @timgdavies did to the 1.0 spec, or if there are plans to do so? Or maybe your instance is unaffected by those changes? I'd be interested at poking around the data so just curious to know in advance any quirks I might find. Thanks!

boxfoot commented 7 years ago

Exciting to see this conversation happening -- these are questions we're also actively exploring at HL. I'm having a hard time catching up on this conversation and what's been done so far.

@kinlane you seem to have several open-referral repositories in your github account -- can you clarify how they fit together and what we should be looking at?

Is there a space where folks are diving into some of the specific questions being raised here?

NeilMcKLogic commented 7 years ago

Hi @kinlane , just sent you some documentation privately for inspiration on the spec. A few things catch my eye:

-Need to be able to search not just by org_name but also service_name -If a Category search can be an AIRS Taxonomy search (by name or by code), that would work. -Probably want to specify a result set sorting preference (alphabetical, proximity, relevance, ....?)

Also up for discussion is what fields appear in the search results for matching records - all? just a subset? related records (some or all) ?

kelfink commented 7 years ago

Hi, wanted to chip in on this conversation with a comment about moving data from CSV files, such as federal records of stores accepting EBT, into Ohana. In our case, we had location info, but not really any organization, program, or service. So we ended up inventing a lot of it. I'm tempted to write a more generic tool that uses algorithms to create HSDS compatible records, so anyone can adapt those. The work we did importing the csv files is from almost a year ago, and needs some updating.
https://github.com/SacHomeless/ohana-client . I'm interested in your views of how to sustainably bring in and maintain such data. Thanks!

kelfink commented 7 years ago

Separate question about Open211 data (maybe the wrong thread?) for the upcoming Open Data day tomorrow. I'd like to gather information about how each California county treats its 211 data, and how that affects the community at large. But I am still searching for metrics our volunteers can gather to show the benefits of the open data. Suggestions? thanks...

NeilMcKLogic commented 7 years ago

hi @kelfink , this probably warrants a separate thread. If you want to provide more specifics I bet I can answer as roughly 30 of the 35 counties in California with 211 service use our software platform, iCarol, for resource database management (and more).

greggish commented 7 years ago

@kelfink it might be good to see a sample of the data you ended up with for EBT locations. it's an interesting edge case.

robredpath commented 4 years ago

We've got an API spec now, so closing!