search

Unidata / netcdf-java

The Unidata netcdf-java library
https://docs.unidata.ucar.edu/netcdf-java/current/userguide/index.html
BSD 3-Clause "New" or "Revised" License
146 stars 71 forks source link

Consider GeoAPI for referencing and metadata aspects of new UCAR API #256

Open desruisseaux opened 4 years ago

desruisseaux commented 4 years ago

Following on the NetCDF-Java - Looking Ahead blog, I propose to use GeoAPI for referencing by coordinates aspects (map projections) and possibly some metadata. Note that I'm not proposing to adopt any particular implementation — GeoAPI is only a set of Java interfaces, implementation can stay the UCAR one. This is an approach similar to JDBC for databases. GeoAPI can be at the hearth of UCAR new API or can be kept isolated as wrappers in a separated module, at your choice.

What is GeoAPI

GeoAPI 3.0.1 is an adopted Open Geospatial Consortium (OGC) standard. It takes the UML models of the following standards and translate them into Java interfaces:

For example map projection can be represented by an instance of ProjectedCRS. This interface provides methods for querying the datum, the axis units, their directions, etc. All those properties are defined in ISO 19111, which has been carefully designed by geodesic experts.

Using GeoAPI would add the following weight to the library:

Benefits

A first benefit of using GeoAPI is to inherit knowledge from the ISO 19111 expert group. Projects designing their own API are at risk of missing important aspects if an accuracy better than a few meters is desired. Referencing by coordinates is both crucial to geospatial libraries and very often an underestimated challenge. Even the popular PROJ.4 library had significant design flaws (fixed in PROJ.6 by adopting the ISO 19111 conceptual model completed by inspiration from GeoAPI).

A second benefit is to give the possibility to plugin alternative implementations if desired. Existing implementations include JNI wrappers around the PROJ.6 library and Apache Spatial Information System (pure Java). If the netCDF library uses CoordinateReferenceSystem interface and CoordinateOperation for transforming coordinates, then the library can perform map projections using its own internal implementation, or using Apache SIS, PROJ.6 or other implementations at user choice and transparently.

A third benefit is that it can enable new services. For example it is possible to use Apache SIS Well Known Text (WKT) parser for reading a coordinate reference system definition and instantiating UCAR map projection objects (not necessarily SIS objects) from that WKT.

Another benefit is that if netCDF, PROJ.6 and Apache SIS API are derived from the same conceptual model (ISO 19111), then the similarities make easier for a developer familiar with one library to master the "referencing by coordinates" aspect of the other library.

Inconvenient

Less control from UCAR on that API (but GeoAPI development is open and any interested OGC members have equal vote). However it is always possible to define custom extensions in implementation classes.

Implementation strategy

Two approaches are possible. The first one is to use GeoAPI interfaces everywhere netCDF API needs to represent a coordinate reference system or a coordinate operation.

A second approach is to keep the core of netCDF library independent of GeoAPI and implement GeoAPI interfaces as wrappers around UCAR netCDF in a separated module.

Existing material

Wrappers already exist as a demonstration project. I would be happy to donate that code if desired; it could be used as a starting point. I can also volunteer for contributing to the work of introducing GeoAPI in the netCDF library if desired, and proposing maintenance when GeoAPI is updated to a new version.

JohnLCaron commented 3 years ago

Hi Martin:

Im looking over your org.opengis.wrapper.netcdf code to see what it looks like, and what changes would be needed to use the new netcdf-java APIs. Our API is extensively rewritten and not backwards compatible. We are using a 6 month major release schedule, and look to release version 6 by the end of the year, and start version 7 after that.

We are considering the possibility of adding a separate geoapi module that wraps netcdf-java to provide an implementation of GeoAPI.

If I understand correctly your package needs org.opengis:geoapi:4.0-SNAPSHOT? Is this available in any maven repository?

Any comments on how complete your package is? Is there a lot of work still to be done to implement GeoAPI? Im looking hopefully at just porting your work to the new netcdf-java library.

Regards, John

desruisseaux commented 3 years ago

Hello John

This is good news! A separated module wrapping new netCDF API looks good to me. I propose to implement GeoAPI 3.0.1, which is current OGC official release and is deployed on Maven Central. In Apache SIS and in PROJ-JNI wrappers I have a separated branch named "geoapi-4.0" which implements GeoAPI 4.0-SNAPSHOT, but I do that only for anticipation. I suggest to omit that branch if too cumbersome. I can help to downgrade the dependency from 4.0-SNAPSHOT to 3.0.1; the changes for netCDF wrappers should be small.

Before to implement GeoAPI interfaces, a first step would be to get an implementation of javax.measure.Unit interface defined by JSR-363 (Units of Measurement). This is the only GeoAPI dependency. I think that UCAR has its own unit system, so you could choose to create javax.measure.Unit wrappers around UCAR implementation. Alternatively you can choose an existing JSR-363 implementation. This is what the current org.opengis.wrapper.netcdf package does for simplicity, but not necessarily what I would recommend. Instead if UCAR invested in implementing a unit of measurement framework it makes sense to use it.

In the meantime I can prepare in a separated repository an org.opengis.wrapper.netcdf package downgraded to GeoAPI 3.0.1 and without the Units class (or with an empty skeleton) if we replace that class with wrappers around UCAR unit system. You could then use that material for adaptations to new UCAR API.

Third step would be to migrate the tests that do not depend on GeoAPI 4.0-SNAPSHOT. After completion, I would suggest to delete the org.opengis.wrapper.netcdf package from GeoAPI repository and redirect to UCAR repository instead.

Does above plan would be okay?

Regards, Martin

JohnLCaron commented 3 years ago

Hi Martin:

  1. Why do you recommend against using "an existing JSR-363 implementation" ?
  2. Downgrading to GeoAPI 3.0.1 seems like the right thing to do.
  3. How complete is the existing org.opengis.wrapper.netcdf towards implementing all of geoAPI?

Reagrds, John

desruisseaux commented 3 years ago

Hello John

It is not exactly a recommendation against using an existing JSR-363 implementation. It seems to me that UCAR library has its own unit system, in which case I assume that using that system may give more consistency with UCAR internal working. If a user gets a unit from UCAR (e.g. from some read operation) and later gave that unit to another UCAR method (e.g. to some write operation), we could avoid the risk of information lost that a "UCAR unit → JSR-363 implementation → UCAR unit" cycle may cause. Instead it would be just wrap/unwrap.

It is also possible to not commit to any JSR-363 implementation, let user choose the JSR-363 implementation of his/her choice and see later if doing wrappers around UCAR unit system is desired. Actually the org.opengis.wrapper.netcdf package is already designed that way.

For question 3, the existing org.opengis.wrapper.netcdf wrappers cover 2 or 3 parts:

Metadata (ISO 19115)

That standard contains about 400 properties. The current wrappers map about 15~20 of them to global attributes. E.g. the identificationInfo/extent/geographicElement/eastBoundLongitude property maps to the ACDD "geospatial_lon_max" global attribute. I think current wrappers provide a good starting point; more mapping can be added progressively in the future.

I think that variable attributes are not yet mapped, but I can help for completing that part.

Referencing (ISO 19111)

Current wrappers are complete enough for instantiating geographic, projected, vertical and temporal Coordinate Reference Systems (CRS) backed by UCAR map projection engine, and for performing coordinate transformations between them. It can be demonstrated by executing a subset of GeoAPI conformance test suite, which verifies the map projection correctness.

Future work

The wrappers contains some work for the support of grid coverages (rasters), but the corresponding GeoAPI interfaces have not been officially released yet. That part would be omitted for an initial release based on GeoAPI 3.0.1 (users can use directly UCAR API in the meantime), but can be kept on a branch for future work.

desruisseaux commented 3 years ago

The Apache Software Foundation (ASF), the Open Geospatial Consortium (OGC) and the OSGeo foundation will do a joint code sprint on February 17 to 19. I proposed an OGC GeoAPI track during that code sprint, with tasks listed in a https://github.com/opengeospatial/joint-ogc-osgeo-asf-sprint-2021/issues/2 comment. Would it be an opportunity for starting this "GeoAPI wrappers for UCAR library" task?

desruisseaux commented 3 years ago

Hello John

I would like to take the opportunity of OGC/ASF/OSGeo code sprint next week for getting the ball rolling. I would like to create a GitHub source code repository and start migrating the current GeoAPI wrappers for UCAR library. What should be the actions?

Maybe a good way to progress would be to have someone from UCAR to register to the code sprint (this is free and open to everyone) and connect to the GeoAPI virtual room sometime during those 3 days?

lesserwhirls commented 3 years ago

Greetings @desruisseaux! I think the code sprint would be a great opportunity to start this work. Let me ask a few others at Unidata if they have plan to attend already, and if not, I'll get signed up. I'm happy to setup a repository within the Unidata organization for this effort. In terms of a package name, not quite sure. Do you have any suggestions?

desruisseaux commented 3 years ago

Hello Lesser, thanks for the reply! For the package name, I suggest something starting with ucar for consistency with existing UCAR API. Maybe ucar.geoapi?

lesserwhirls commented 3 years ago

Greetings @desruisseaux - it looks like I missed the deadline to register (sorry about that). Would I still be able to drop into the GeoAPI virtual room?

desruisseaux commented 3 years ago

I think so. Let me send an email to the organizer.

desruisseaux commented 3 years ago

Hello all. The GeoAPI channel is open from today to Friday inclusive: https://meet.jit.si/GeoAPI

If anyone is curious about what are the other channels, the complete list is here.

lesserwhirls commented 3 years ago

Thank you @desruisseaux! I'm going to drop in the GeoAPI virtual room today at 1:30 EST.

lesserwhirls commented 3 years ago

For those following along, the initial work on this effort will be done over at https://github.com/Unidata/geoapi-netcdf-java.

Here is a summary of the next steps (much more detail above):

desruisseaux commented 3 years ago

Hello @lesserwhirls . If you have a chance, could you connect to the Jitsi room at about 16:00 EST (30 minutes before the brief in the main code sprint room) for a quick update on current state?

lesserwhirls commented 3 years ago

Sure thing, but I will only be able to talk via chat as I will be in another meeting from 15:00 - 17:00 EST. Will take be ok?

desruisseaux commented 3 years ago

Yes no problem. We can also postdone to tomorrow if more convenient.

desruisseaux commented 3 years ago

Hello @lesserwhirls . If you have 10 minutes available between 15:00 and 15:30 EST (the end of the code sprint), I could share my screen for a small demo and for showing some key points in the code. If you do not have time, no problem.

lesserwhirls commented 3 years ago

I'm so sorry Martin, I'm just now seeing your message. I had planned on trying to join the room today but I could not make my schedule work. I'd love to meet up sometime next week if you have the time / availability.

desruisseaux commented 3 years ago

Sure just let me know when you will be available (no rush). If the https://meet.jit.si/GeoAPI is not open any more, we will create another Jitsi room on another server.

desruisseaux commented 3 years ago

Hello @lesserwhirls . Do you have some availability this week or later for a short presentation (~10 minutes) on what the wrappers can do? During the Joint ASF-OGC-OSGeo code sprint, I presented a demo opening a netCDF file using UCAR library and formatting its Coordinate Reference System (CRS) in Well Known Text (WKT) format using Apache SIS.

lesserwhirls commented 3 years ago

Hello @desruisseaux! I do have time this week and next to meet you for a demo. What would be your preferred method for coordinating a time?

desruisseaux commented 3 years ago

Thanks for the merge! I will update GeoAPI web site tomorrow for redirecting to UCAR. For a demo, any day of your choice except Thursday is fine for me, if possible sometime between 11:00 and 16:00 Pacific time (I presume this is your time zone?).

lesserwhirls commented 3 years ago

I am in Mountain time (currently UTC-6. In which time zone are you located? How does 12:00 MDT (18:00 UTC) on Friday, March 19th look for you? I'm happy to create a Google Meet for us to use, or we can use another platform if you'd like.

desruisseaux commented 3 years ago

Friday 18:00 UTC is perfect for me. This is 19:00 in my time zone. Google meet is fine for me.

lesserwhirls commented 3 years ago

Excellent - I've just sent out an invite to your geomatys email address for this Friday. Is it ok with you if I invite a few others from Unidata (and possibly @JohnLCaron if he is available?)?

desruisseaux commented 3 years ago

Sure!