Display grants data (& decide on an API convention)

[beechnut]

Context We're getting ready to start sending real (sample) data from the API to the frontend.

As we begin displaying grants data from the API on the frontend, the question is raised: what is the shape of the response we return? #389 is only scoped to provide "minimal JSON".

Considerations include:

• Format (e.g. JSON:API, JSON-LD, HAL, Collection+JSON)
• Navigation / hypermedia
• Versioning (URL parameter, Content-Type headers)
• (Maybe) How search works

We'll know we're done when:

• [ ] We've decided on the API design for /grants
• [x] We have an initial idea of how versioning will work
• [x] We're displaying a list of grants from the database

[beechnut]

Here's what I'm thinking so far.

Versioning

First, use an "always add, never remove" versioning strategy.

For specifying a version, allow both:

• A Content-Type header (e.g. Accept: application/vnd.api+json; gov.usda.fs.ga.api.version=1)
• A URL component (one of: a subdomain like v2.api.ga.fs.usda.gov or a base path api.ga.fs.usda.gov/v2/) to set the version.

Clients, including our Angular app, can use a header to not complicate the URL string. And if developers want to do some manual testing, they can use the version in the URL string, since that's easier in a browser than setting a header.

API Standard

The pitfalls of inventing one's own standard for something as standardizable as HTTP client-server communication are many, so adopting a standard precludes wasting significant amounts of time and money on the design of those interactions.

The JSON:API specification comes out of work done by some of the top thinkers in modern web development, including Yehuda Katz, Dan Gebhardt, and Steve Klabnik. The specification covers both the request and response parts of the cycle, relationships, links, record updates and deletion, how to respond to long-running jobs, and more.

We can minimally implement the JSON:API standard with smaller code changes now, and for robust implementation, eventually adopt a few of the abundant and well-maintained libraries for both server-side and client-side.

Questions

• [ ] @AdamUSDA @jayrbergjr: Will users of the API have to register to be able to use the API?
• [ ] @AdamUSDA @jayrbergjr @neilmb: Should we write these up as ADRs? I think writing up versioning strategy would be important because versioning affects how change will be handled over the lifetime of the API. I'm not convinced we need to ADR the standard, because if the standard needs to change in future versions, it can without breaking anything, so long as we're following our versioning strategy.

[AdamUSDA]

Hey @beechnut the API will more than likely use a OATH token or URL mapping to access the API. The API gateway can authenticate the call using the URL and then make the Lambda request so there will be no need to register the API.

When we integrate with Mulesoft external customers they will be able to register themselves for the public API's but that's a much more robust process and outside the scope of this POC. I don't think that we need to dive into that process for the POC.

When we integrate with Mulesoft our goal will be to gain access to the DBs in DISC using an APIs. The DBAs will create an API that will return a SQL results from the Database in a JSON format. That will probably require a authentication process with Mulesoft but I'm not familiar with how that is done at this time. It maybe a URL mapping but I don't really know. I've asked for access to the Dev Mulesoft environment.

I'm trying not to add to much process to the POC API process but more than likely we will use a Serverless framework like this when we get everything up and running. (https://www.serverless.com/framework/docs/). That can provide API versioning using GitHub. I'm not certain if that's what your asking.

Using the Serverless framework will allow use to build and deploy our serverless code from GitHub and Jenkins. It's very robust but adds even more complexity to the POC which again may not be needed. That would require even more research on your part Matt but I not certain to what level of fidelity we want to go in the POC?

[beechnut]

@AdamUSDA Thanks for the info regarding registration / authorization, and that helps to know where Mulesoft fits into all of this.

It looks like we're talking about different uses of the word "versioning", which makes sense because both of the things we're talking about are "versioning". I think what you're talking about is versioning the codebase, deployments, and the like. This issue is meant to be about the strategy for versioning the public interface of the API we're developing. The core question we're trying to address with this issue is: how will the team handle it when the developers eventually have to alter the API in a way that breaks existing client implementations? Hopefully this adds context to the above Versioning section.

What this would mean in practice is that clients—including our own Angular app—will specify a version using either a version number in the Accept HTTP header (essentially Accept: {media type here}; version=1) or a URL path (/v1/grants, /v2/users).

When the time inevitably comes that developers need to make breaking changes to the API—that is, public-facing changes that would break existing client implementations—developers would publish that as a new API Version, e.g. v2. Existing clients can continue to use v1 without breaking, and when they're ready to upgrade, can gracefully transition to v2 (or continue using v1 for the life of the API if they so choose).

I asked about registration because if we'll want to eventually "turn off" versions, we'd need a way to communicate to every developer using v1, so some registration process would be necessary to collect contact information. If there's no registration, then a strategy of "always add, never remove" versions would promote the greatest long-term success with client implementations.

Using the JSON:API standard also sets the project up well to avoid needing to produce breaking changes as long as possible, since the standard prescribes server<->client interaction based on years or decades of research and practical experience with HTTP. There will probably come a time when breaking changes and a new version are needed, but I'm taking it as a good sign that I can't currently forecast or imagine what would drive that need, if this standard is used.

[AdamUSDA]

Hey @beechnut . Thanks for the clarification.

The projects that I've worked on have used URL path /v1/grants, /v2/grants model but I'm really agnostic. If the header model has benefit over the URL model I'm open to exploring that option. If not let's go with the URL path.

To be honest I'm not certain if AWS API Gateway has registration capabilities. I know that Mulesoft does but I'm not certain if AWS does. Since we control both sides of the equation I assume that we can manage those transitions gracefully in both situations.

As you can imagine it gets much more complicated when we provide access to external partners. We can use the AWS model and never depreciate anything or use the google model and be one email away from a very interesting day because they actively depreciate APIs with little to no notice. Knowing how things are around here I assume that we will use the AWS model.

Does that answer your questions?

[beechnut]

@AdamUSDA Thanks, it definitely does! And the code I've written in #403 allows the client to choose whether they use the URL or the header. I also expect most to use URL, but I suspect we'll get API consumers (including our Angular app) that appreciate the option of using a header.