Inserting new oil-type temporarily or permanentely from library user

[gauteh]

Hi,

it would be very useful for us (https://github.com/OpenDrift/opendrift) if it was possible to temporarily add some oils to the database. We often need to add custom new oils, and do not have time to wait for a new release of the oil library before it needs to be used. It would still be great if these oils can be merged in eventually, but I think this is the only issue that requires us to keep a separate fork at the moment.

Best regards, Gaute (opendrift-developer)

cc: @knutfrode

[ChrisBarker-NOAA]

It's time for a conversation:

We are well on our way to an updated Oil Database code that will make all this easier -- among other things, by separating out the data itself from the code that works with the data.

The code is not up on gitHub yet, but we could do that sooner than later if you want to collaborate.

You can see the new UI here:

adios-stage.orr.noaa.gov

If you look at an oil record and download it, you will get a JSON representation -- which will be our future exchange format.

We also have made anew "oil" object in our GNOME software, more suitable for computation, and are just starting on the code to build a "GNOME OIL" from the new oil data model.

Anyway -- it would be great to collaborate on. this.

[knutfrode]

Hi Chris,

The UI seems brilliant. We are definitely interested to collaborate on this.

The first two things that come into mind:

• A missing possibility so far has been to provide a maximum water uptake for each oil. This seems to have been hardcoded as 90% for all oils.
• For an operational perspective, we (and probably others) would prefer to not be dependent on the database to be available online. Thus a possibility to export/download a subset would be useful.

[ChrisBarker-NOAA]

Sounds good, as to your points:

• A missing possibility so far has been to provide a maximum water uptake for each oil.

That's a real challenge, that's out of scope of this project :-(. -- but I think we will no longer put that in the database, as it's a guess anyway. It will be up to the model to decide what to do.

However, there are some records with data about lab-emulsified oils, that could be used to populate that field.

I'll try to get a copy up on gitHub soon so you can take a look.

[knutfrode]

Thats understandable, limiting the scope is a prerequisite for getting things done :-)

It might however be useful if there is a possibility to add "user defined" fields (metadata key-value pairs) which are not necessarily mirrored in the database. Then these could be used to store such kind of information with user-based conventions, allowing for more flexibility. But anyway, when code comes to GitHub, we can see if there is something we could possibly contribute.

[ChrisBarker-NOAA]

Indeed -- that is exactly why we are using JSON as an exchange format, and, in fact, using a "NoSQL" database on the back end (MongoDB).

[gauteh]

This looks very interesting, makes a lot of sense (almost obvious in retrospect) to keep the data separate in a way like this. Is the code you are talking about a python API + methods for computing additional parameters for this data repository? If the same python-object can be instantiated from a local JSON file or dictionary then it would be possible to provide a small offline backup database, as well as new oils that may not yet have been accepted into the database. Depending on how things are set up this might already be possible?

[ChrisBarker-NOAA]

@gauteh: yes, that's the idea.

We have a Python object that can load and save from/to JSON. This object is designed around storing the data in a structured way for display and querying (with units, etc).

We are now getting started on adding code to make it easy to extract the actual numbers for computational purposes, and estimate missing parameters, etc. Much of this is in the existing lib, we just need to refactor it a bit.

[knutfrode]

This sounds all good.

To check if I understand correctly: The present ADIOS OilLibrary consists of both a database and some algorithms for calculating derived oil properties. Then PyGNOME contains other ("higher level") algorithms which utilize both the ADIOS algorithms as well as the ADIOS oil data?

But in the future, ADIOS will only contain the data, and all algorithms will be contained in the PyGNOME software?

[ChrisBarker-NOAA]

Amost: we see the division like this:

GNOME has an GnomeOil object that has everything it needs to be used, at run time, in the model.

adiosdb has everything needed to manage the data itself.

The GnomeOil has the Pseudo-components required to run the model, so we need a bunch of code that takes the data in the database, and builds a GnomeOIl out of it. That code will (at least for now) go in the adiosdb package -- so you will be able to load an oil record from JSON, or from our database server, or ???, and make a GnomeOIl out of it (actually the JSON required to make GnomeOIl.

I'm hoping that other groups might contribute code to make an oil object suitable for other models, e.g. OSCAR, or SIMAP or COSMOS or OpenOil :-)

If it turns out that some of the code we use in our GnomeOIl is useful to you, then we. can see about putting that code some where we can share it without you needing all of PYGNOME. -- maybe in the adiosdb package, maybe in a third package.

[knutfrode]

Thank you for clarifying.

[ChrisBarker-NOAA]

We've got the new code up on gitHub now:

https://github.com/NOAA-ORR-ERD/adios_oil_database

You probably want to look at the develop branch, we are actively working on it.

We need to do some cleanup, but the core code should be pretty stable

There are Sphinx docs in the adiso_db documentation dir -- but I"m having trouble getting gitHub to publish them, so you'll need to run make html yourself to build them.