GNIP: Handling raster mosaics

[jj0hns0n]

Overview

GeoServer supports mosaics with custom dimensions, but there is no way to configure them on GeoNode.

It is proposed to add new (optional) functionalities to provide handling of mosaics.

Operations shall include as a minimum:

• allow users to create a mosaic with custom dimensions (or at least a simplified dimensions set, including a couple of time dimensions and an elevation dimension).
• allow users to add granules to a mosaic.
• allow users to remove granules from a mosaic.

Such mosaics should be recorded as Layers inside GeoNode.

Proposed By

• Emanuele Tajariol
• Alessio Fabiani

Filed by Jeffrey Johnson

Assigned to Release

TBD

State

For Review and Comment

Motivation

GeoNode does not currently provide any way for configuring mosaics.

Proposal

Model

The Layer class should be modified so that it include information if the given instance is referring to a mosaic, and some related info that need to be persisted.

Configuration

In the same way the time info is configured for vector layers, a new option will be added to the UPLOADER.OPTIONS dict:

UPLOADER = {
    'BACKEND': 'geonode.importer',
    'OPTIONS': {
        'TIME_ENABLED': True,
        'MOSAIC_ENABLED': True,

When a GeoTIFF is selected for upload, if MOSAIC_ENABLED is true, we'll have a checkbox "Mosaic" (just like the "Time" checkbox for vector layers) to request that the flow for the mosaic handling should be started.

Use cases

If the checkbox "Mosaic" is checked, once the data upload is completed, the user will be asked if the uploaded file is a granule belonging to an existent Mosaic, or if a new Mosaic should be created.

We have 2 different flows:

• the user just uploaded the first granule for a brand new imagemosaic layer:
  1. upload the geotiff file.
  2. ask if the file represents a granule for a brand new mosaic (yes).
  3. (a) create the mosaic (setup the dimensions).
  4. setup the granule (assign values to the dimensions).
  5. setup and ingestion in geoserver.
• the uploaded file is a new granule to add to an existing imagemosaic layer:
  1. upload the geotiff file
  2. ask if the file represents a granule for a brand new mosaic (no)
  3. (b) choose an existing imagemosaic layer in geonode
  4. setup the granule (assign values to the dimensions)
  5. setup and ingestion in geoserver

     Creating the mosaic

When creating the mosaic, the user should give information about the associated dimensions, if any.

GeoServer uses a quite complex set of files to determine how to retrieve the custom dimensions, extracting them from the granule file name. We want GeoNode to create automatically these files for the users. Setting up these information is anyway quite complex, so the first proposed implementation will offer only a limited choice of additional dimensions: the user will be able to add up to:

• 2 time dimensions
• 1 spatial dimension (elevation).

The combinations of this dimension set should anyway cover a great number of use cases.

The user will be able to choose one of these format for each of the time dimension:

• year
• year + month
• year + month + day
• year + month + day + hour + minute + second

The elevation will always be parsed as a decimal number.

Once the mosaic is created, the user will be redirected to a page to enter the dimension values of the granule just uploaded.

Setting up a granule

If the mosaic has one or more dimensions, the user will have to insert in the GUI the related values for each of them. GeoNode should already know about the number and the format of the associated dimensions, so the input form should be built appropriately.

We may introduce here a bit of heuristic to find out if the file name of the uploaded file already brings any information about the dimensions, and pre-fill the GUI fields with them.

As said, GeoServer parse the granule file name to retrieve the dimension values. By limiting the dimension set, and by requesting to edit the dimension values in the UI, we are relieving the users from the burden of setting up and testing the regular expressions needed for parsing the input files names.

Ingesting granules in GeoServer

This stage receives as input the granule file, the dimensions description, the dimension values and a reference to the imagemosaic layer. The logic will format the filename according to what GeoServer is expecting, and will make the proper calls to the GeoServer importer module (that's why the importer backend should be set in the configuration).

The importer module has received many improvements in GeoServer 2.7 (see http://docs.geoserver.org/stable/en/user/extensions/importer/rest_examples.html#adding-a-new-granule-into-an-existing-mosaic), so we're going to leverage on these new features, such as image preprocessing.

Deleting granules

The "Update Layer" dialog will have the options "Manage granules" and "Delete".

The "Delete" button will maintain the same semantic as for the other objects, and will remove the whole imagemosaic Layer.

The "Manage granules" button will display a page containing a list of all the granules of the mosaic, where the single granules can be removed from the mosaic.

In order to update a single granule, the user have to delete it from this list, and then run the whole upload procedure.

Issues and future improvements

GeoServer version

GeoServer 2.7.x implements improved functionalities to deal with mosaics. This proposal depends on the migration to GeoServer 2.7 (See https://github.com/GeoNode/geonodeissues/2040).

Batch import

The described flows only explain how to add granules one by one. In the future we may want to enable to user to upload and ingest a batch of granules.

File formats

The imagemosaic module in GeoServer can work with several types of raster formats. For the sake of semplicity of the first implementation, only GeoTIFF will be supported initially.

Image preprocessing

Preprocessing on GeoServer side may be made optional, in case the provided GeoTIFF are just ready to be published.

Testing

This module should provide its own unit tests, and any integration tests should be added to maintain or increase coverage as necessary.

Alternatives

Feedback

See the discussion on the geonode-dev list here:

http://lists.osgeo.org/pipermail/geonode-devel/2015-July/000162.html

Voting

N/A

[jj0hns0n]

This would be a fantastic feature!

Couple of comments:

• Can we see some idea of what the model changes would look like. What changes would be made to Layer and what can stand in its own Mosaic class.
• Where will the views etc go that drive this. Do we need a new contrib module for this?
• Can we see a mockup/wireframe of what this form/flow would look like for the Mosaic options when uploading
• What changes are needed on the python side of the uploader to handle for this kind of functionality. Our implementation is quite out of date @ischneider was the last one to touch it quite a while ago. See the
• Can you flesh out the idea of a batch importer and how that would work. I believe that the mapstory folks did some work in this area as well. @cholmes may know more

[jj0hns0n]

One other question. How does geonode currently handle mosaics that are configured manually in GeoServer?

[jj0hns0n]

@afabiani what is the status on this. Ready to be merged after 2.4?

[afabiani]

almost ready but I am not able to merge it for 2.4 release. I need more time to test it well and be sure to not break anything.

[jj0hns0n]

@afabiani whats the status on this work?

[afabiani]

Pull Request relative to this here

https://github.com/GeoNode/geonode/pull/2460

Best Regards, Alessio Fabiani.

GeoServer Professional Services from the experts!

Visit http://goo.gl/it488V for more information.

Ing. Alessio Fabiani @alfa7691 Founder/Technical Lead

GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) Italy phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 331 6233686

http://www.geo-solutions.it http://twitter.com/geosolutions_it

---

AVVERTENZE AI SENSI DEL D.Lgs. 196/2003

Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003.

The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc.

---

On Tue, Apr 5, 2016 at 5:18 PM, Jeffrey Johnson notifications@github.com wrote:

@afabiani https://github.com/afabiani whats the status on this work?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2211#issuecomment-205854397

[jj0hns0n]

@afabiani this can be closed now that this is merged right? Lets make sure we do some final careful testing before the 2.5

[afabiani]

Yes. It would be better to test it again after the latest commits on GeoNode just to be sure anything has been broken.