Automatic indexing of DG Open Data imagery

[cgiovando]

Every time Digital Globe releases new imagery in https://www.digitalglobe.com/opendata/it still requires a very manual process to add that imagery in OAM and make it available as TMS/WMTS. This process can significantly delay access to imagery for HOT/OSM mappers, and sometime make it impossibile to access at all (when someone is not available to manually review, process and upload the imagery in OAM).

For reference these are the steps involved:

• browse https://www.digitalglobe.com/opendata and review images for a specific event using the Preview function. This allows anyone with a web browser to look at the images and understand whether they are usable with regards to cloud cover, but does not provide any other metadata information or spatial context.
• with the specific scene ID (CATID) from that list, one can go to https://discover.digitalglobe.com and look up more details about that image, including metadata, a more detailed overview and its footprint over a reference map.
• another option to review each scene is to download the individual tiles provided as GeoTIFF files in the Open Data portal and then compose them into a desktop GIS software. This process requires large bandwidth, as each scene is made of 10-20 tiles of approximately 1.3GB each (GeoTIFF + overview)
• in order to avoid downloading imagery over an office or home network, it is more convenient to spin up a cloud instance (AWS, Azure, etc) with enough storage and computing capacity to review and process these images. The transfer of data from the DG Open Data Portal to the cloud instance is much faster, as well the subsequent upload to OAM.
• once all tiles for one scenes are downloaded and deemed useful (low cloud cover, acceptable nadir angle, etc), then a combination of GDAL utilities (gdalbuildvrt, gdal_translate, gdaladdo) allows to mosaic them together into a single GeoTIFF. Some manual steps may be required through a visual desktop app like QGIS to interactively select subsets of each scene (e.g. to to remove imagery over ocean or completely cloudy).
• depending on the size of the final mosaicked scene, it may be required to split each scene into two or more separate sub-scenes to avoid any limitation in file size when uploading to OAM. Another option is to output the mosaicked image using a compression algorithm (e.g. JPEG)
• with the scene ready on the cloud instance disk as a single GeoTIFF, one can then upload to OAM through the interactive web uploader (https://upload.openaerialmap.org) or directly via its API.

As you can see, this is a very manual and lengthy process that could be partially or entirely automated. Here are some suggestions - please add yours in the comments.

• Images made available by DG in the Open Data portal should not have to be downloaded and manually uploaded into OAM. The Open Data portal should be configured to be an OIN node that indexes imagery directly to OAM, and for any other use (e.g. the portal's web page itself with direct download links). Assuming license compatibility, this would be the most efficient way to make provider's imagery accessible to all users via OAM, TMS, and WMTS for humanitarian response.
• An on demand worker (a temporary cloud instance) could be configured to run scripts that automatically download, process and upload imagery. This would be the automated version of the manual process above, but without any review step. In this case all imagery made available by DG will be posted in OAM, even if cloudy or not usable for other reasons. No preview/review is involved, potentially wasting resources and storage.
• DG could provide a compressed version of each scene, rather than full resolution GeoTIFFs split into multiple tiles. For most humanitarian response applications JPEG compression is perfectly suitable and would make images much easier to access.
• Cloud Optimized GeoTIFF (http://www.cogeo.org) should be implemented by providers to make their imagery easier to handle by web processing services, dynamic tilers, etc.

The above considerations are in this case referred to DG's open data program, but can be applied to any other provider who wants or is already publishing imagery under an open license for humanitarian response.

[bgirardot]

For what it is worth, DG is hand selecting imagery for the Open Data portal so it is all generally useful. Other blocker here is OIN does not support the license DG releases under, but it should/could. It is also fine for OAM to support DG's license and pull the imagery from DG, no need for DG to be an OIN node which is not very well developed at this time.