search

NOAA-ORR-ERD / adios_oil_database

System for managing petroleum, product data for use in oil spill response
https://adios.orr.noaa.gov
6 stars 1 forks source link

Versioning of the API and the oils #4

Open gauteh opened 3 years ago

gauteh commented 3 years ago

Hi,

once the API is stable do you plan on versioning it so that we can rely on our client code always working for the url? e.g. https://adios.orr.noaa.gov/oils/v1/.. etc? A couple of other concerns from us, removing an oil from the database would be considered an API breakage for our own services. Do you have any thoughts about how that would happen? Can we rely on oils never disappearing from the list (at least if it is behind a versioned URL)? (adding new oils is not considered breaking by us).

Technically, modifying an oil would perhaps also be considered breaking, but we have not practiced this very strictly. Versioning oils would perhaps be a solution (e.g. https://adios.orr.noaa.gov/oils/NO0001/v1/). Say if you want to reproduce a simulation and oil-properties have changed in the meantime.

Regards, Gaute

ChrisBarker-NOAA commented 3 years ago

We weren't really anticipating any real changes to the API itself (other than adding features) -- it's pretty simple. But yes, versioning it would be good. We may wait for a breakinging change before we actually adda version to the API. But we're open to ideas on that.

As for the data itself:

We are managing the data in a git repo -- a big collection of JSON files. (we do intend to publish that repo on gitHub)

So on a low level everything is fully versioned. However, how to expose the versions via the API is another question. I don't think we'll ever fully expose it -- as in making all old versions of the data available. THe API will likely only serve the latest version of the data.

At this point, we are actively validating and cleaning up data, so haven't really versioned the database -- too many small changes -- we don't want to wait for a "release" to put even a single small correction in a single oil on the live version. However, we do want SOME kind of version info -- so we are going to probably add a "last edited" date to the API.

As for removing oils: my first thought is that we won't remove anything. However, we have been looking over some of the old data, and there appear to be some duplicate records, so we may in fact a remove a few of those. I suppose we could put aliases in or something -- have to think about that.

As for versioning individual records: That's probably a good idea, so we can know for sure what's changed. Ideas about how to do that:

Probably some combination of the above :-)

However, if you want to be able to exactly reproduce a simulation, I would suggest saving the JSON of the oil record itself, along with any configuration you are saving anyway. That's what we do with GNOME: A GNOME "save file" has everything it needs to re-run a simulation.

gauteh commented 3 years ago

Thank you for the answer. We discussed a bit how to deal with to maintain API stability for our services. Changing an oil would probably not require versioning for our part. As you suggest, storing the oil metadata is the way to go, and we will do this for our results too. Removing an oil would be a breaking change, but I think that the best solution for us is to ship the JSON files (or possibly just a subset) in some way as well.

For other types of use we will let users provide oil metadata by JSON as an option, so that they could manually fetch their oil from ADIOS if they want.

It is not a problem for us to scrape the database through the REST-api, but once that JSON repo is available it would probably make sense for us to build the database from there (then we can fix our collection to a version there, and easily detect removed oils).

Keeping the git-commit of the last change on that record would probably be useful, since it would then be possible to restore it easily. Maybe not a very often used feature. Both hashing the data (and storing the git commit of the last edit) might be a bit tedious to do technically.

Again; as long as we just go straight for the JSON-repo or scrape the database all of these issues fall away for our part.

ChrisBarker-NOAA commented 3 years ago

" it would probably make sense for us to build the database from there (then we can fix our collection to a version there, and easily detect removed oils)."

Exactly -- that's why we are maintaining the data in git otherwise, we'd have had to write a bunch of code for all this versioning of the data. And git is actually a pretty good fit for this.

I'll get the repo up soon -- just want to clean it up a bit.