search

GenericMappingTools / gmtserver-admin

Cache data and script for managing the GMT data server
GNU Lesser General Public License v3.0
7 stars 3 forks source link

Explore grdcut to tile images #220

Open Esteban82 opened 10 months ago

Esteban82 commented 10 months ago

Testing tiling images for the server:

Paul, this failed: gmt grdcut @earth_day_30s_p -R0/15/15/30 -Gonetile.tif=gd

grdcut [ERROR]: gmt_gdalwrite: Output driver gd not recognized
grdcut [WARNING]: The following format drivers are configured and support output:
grdcut [WARNING]:   VRT: Virtual Raster
grdcut [WARNING]:   GTiff: GeoTIFF
grdcut [WARNING]:   COG: Cloud optimized GeoTIFF generator
grdcut [WARNING]:   NITF: National Imagery Transmission Format
grdcut [WARNING]:   HFA: Erdas Imagine Images (.img)
grdcut [WARNING]:   ELAS: ELAS
grdcut [WARNING]:   AAIGrid: Arc/Info ASCII Grid
grdcut [WARNING]:   DTED: DTED Elevation Raster
grdcut [WARNING]:   PNG: Portable Network Graphics
grdcut [WARNING]:   JPEG: JPEG JFIF
grdcut [WARNING]:   MEM: In Memory Raster
grdcut [WARNING]:   GIF: Graphics Interchange Format (.gif)
grdcut [WARNING]:   FITS: Flexible Image Transport System
grdcut [WARNING]:   XPM: X11 PixMap Format
grdcut [WARNING]:   BMP: MS Windows Device Independent Bitmap
grdcut [WARNING]:   PCIDSK: PCIDSK Database File
grdcut [WARNING]:   PCRaster: PCRaster Raster File
grdcut [WARNING]:   ILWIS: ILWIS Raster Map
grdcut [WARNING]:   SGI: SGI Image File Format 1.0
grdcut [WARNING]:   SRTMHGT: SRTMHGT File Format
grdcut [WARNING]:   Leveller: Leveller heightfield
grdcut [WARNING]:   Terragen: Terragen heightfield
grdcut [WARNING]:   GMT: GMT NetCDF Grid Format
grdcut [WARNING]:   netCDF: Network Common Data Format
grdcut [WARNING]:   HDF4Image: HDF4 Dataset
grdcut [WARNING]:   ISIS3: USGS Astrogeology ISIS cube (Version 3)
grdcut [WARNING]:   ISIS2: USGS Astrogeology ISIS cube (Version 2)
grdcut [WARNING]:   PDS4: NASA Planetary Data System 4
grdcut [WARNING]:   VICAR: MIPL VICAR file
grdcut [WARNING]:   ERS: ERMapper .ers Labelled
grdcut [WARNING]:   JP2OpenJPEG: JPEG-2000 driver based on OpenJPEG library
grdcut [WARNING]:   FIT: FIT Image
grdcut [WARNING]:   GRIB: GRIdded Binary (.grb, .grb2)
grdcut [WARNING]:   RMF: Raster Matrix Format
grdcut [WARNING]:   WMS: OGC Web Map Service
grdcut [WARNING]:   RST: Idrisi Raster A.1
grdcut [WARNING]:   INGR: Intergraph Raster
grdcut [WARNING]:   GSAG: Golden Software ASCII Grid (.grd)
grdcut [WARNING]:   GSBG: Golden Software Binary Grid (.grd)
grdcut [WARNING]:   GS7BG: Golden Software 7 Binary Grid (.grd)
grdcut [WARNING]:   R: R Object Data Store
grdcut [WARNING]:   KMLSUPEROVERLAY: Kml Super Overlay
grdcut [WARNING]:   WEBP: WEBP
grdcut [WARNING]:   PDF: Geospatial PDF
grdcut [WARNING]:   Rasterlite: Rasterlite
grdcut [WARNING]:   MBTiles: MBTiles
grdcut [WARNING]:   CALS: CALS (Type 1)
grdcut [WARNING]:   WMTS: OGC Web Map Tile Service
grdcut [WARNING]:   MRF: Meta Raster Format
grdcut [WARNING]:   PNM: Portable Pixmap Format (netpbm)
grdcut [WARNING]:   PAux: PCI .aux Labelled
grdcut [WARNING]:   MFF: Vexcel MFF Raster
grdcut [WARNING]:   MFF2: Vexcel MFF2 (HKV) Raster
grdcut [WARNING]:   BT: VTP .bt (Binary Terrain) 1.3 Format
grdcut [WARNING]:   LAN: Erdas .LAN/.GIS
grdcut [WARNING]:   IDA: Image Data and Analysis
grdcut [WARNING]:   LCP: FARSITE v.4 Landscape File (.lcp)
grdcut [WARNING]:   GTX: NOAA Vertical Datum .GTX
grdcut [WARNING]:   NTv2: NTv2 Datum Grid Shift
grdcut [WARNING]:   CTable2: CTable2 Datum Grid Shift
grdcut [WARNING]:   KRO: KOLOR Raw
grdcut [WARNING]:   ROI_PAC: ROI_PAC raster
grdcut [WARNING]:   RRASTER: R Raster
grdcut [WARNING]:   BYN: Natural Resources Canada's Geoid
grdcut [WARNING]:   ARG: Azavea Raster Grid format
grdcut [WARNING]:   USGSDEM: USGS Optional ASCII DEM (and CDED)
grdcut [WARNING]:   BAG: Bathymetry Attributed Grid
grdcut [WARNING]:   NWT_GRD: Northwood Numeric Grid Format .grd/.tab
grdcut [WARNING]:   ADRG: ARC Digitized Raster Graphics
grdcut [WARNING]:   BLX: Magellan topo (.blx)
grdcut [WARNING]:   PostGISRaster: PostGIS Raster driver
grdcut [WARNING]:   SAGA: SAGA GIS Binary Grid (.sdat, .sg-grd-z)
grdcut [WARNING]:   XYZ: ASCII Gridded XYZ
grdcut [WARNING]:   HF2: HF2/HFZ heightfield raster
grdcut [WARNING]:   JPEGLS: JPEGLS
grdcut [WARNING]:   ZMap: ZMap Plus Grid
grdcut [WARNING]:   SIGDEM: Scaled Integer Gridded DEM .sigdem
grdcut [WARNING]:   GNMFile: Geographic Network generic file based model
grdcut [WARNING]:   GNMDatabase: Geographic Network generic DB based model
grdcut [WARNING]:   ESRI Shapefile: ESRI Shapefile
grdcut [WARNING]:   MapInfo File: MapInfo File
grdcut [WARNING]:   S57: IHO S-57 (ENC)
grdcut [WARNING]:   DGN: Microstation DGN
grdcut [WARNING]:   Memory: Memory
grdcut [WARNING]:   CSV: Comma Separated Value (.csv)
grdcut [WARNING]:   GML: Geography Markup Language (GML)
grdcut [WARNING]:   GPX: GPX
grdcut [WARNING]:   LIBKML: Keyhole Markup Language (LIBKML)
grdcut [WARNING]:   KML: Keyhole Markup Language (KML)
grdcut [WARNING]:   GeoJSON: GeoJSON
grdcut [WARNING]:   GeoJSONSeq: GeoJSON Sequence
grdcut [WARNING]:   Interlis 1: Interlis 1
grdcut [WARNING]:   Interlis 2: Interlis 2
grdcut [WARNING]:   OGR_GMT: GMT ASCII Vectors (.gmt)
grdcut [WARNING]:   GPKG: GeoPackage
grdcut [WARNING]:   SQLite: SQLite / Spatialite
grdcut [WARNING]:   WAsP: WAsP .map format
grdcut [WARNING]:   MSSQLSpatial: Microsoft SQL Server Spatial Database
grdcut [WARNING]:   PostgreSQL: PostgreSQL/PostGIS
grdcut [WARNING]:   MySQL: MySQL
grdcut [WARNING]:   DXF: AutoCAD DXF
grdcut [WARNING]:   FlatGeobuf: FlatGeobuf
grdcut [WARNING]:   Geoconcept: Geoconcept
grdcut [WARNING]:   GeoRSS: GeoRSS
grdcut [WARNING]:   GPSTrackMaker: GPSTrackMaker
grdcut [WARNING]:   PGDUMP: PostgreSQL SQL dump
grdcut [WARNING]:   GPSBabel: GPSBabel
grdcut [WARNING]:   CouchDB: CouchDB / GeoCouch
grdcut [WARNING]:   Cloudant: Cloudant / CouchDB
grdcut [WARNING]:   ODS: Open Document/ LibreOffice / OpenOffice Spreadsheet 
grdcut [WARNING]:   XLSX: MS Office Open XML spreadsheet
grdcut [WARNING]:   Elasticsearch: Elastic Search
grdcut [WARNING]:   Carto: Carto
grdcut [WARNING]:   AmigoCloud: AmigoCloud
grdcut [WARNING]:   Selafin: Selafin
grdcut [WARNING]:   JML: OpenJUMP JML
grdcut [WARNING]:   VDV: VDV-451/VDV-452/INTREST Data Format
grdcut [WARNING]:   GMLAS: Geography Markup Language (GML) driven by application schemas
grdcut [WARNING]:   MVT: Mapbox Vector Tiles
grdcut [WARNING]:   NGW: NextGIS Web
grdcut [WARNING]:   MapML: MapML
grdcut [WARNING]:   TIGER: U.S. Census TIGER/Line
grdcut [WARNING]:   ENVI: ENVI .hdr Labelled
grdcut [WARNING]:   EHdr: ESRI .hdr Labelled
grdcut [WARNING]:   ISCE: ISCE raster
grdcut [WARNING]:   Zarr: Zarr
Esteban82 commented 10 months ago

These two worked:

gmt grdcut @earth_day_30s_p.tif -R0/10/0/10 -Gonetile.tiff=GTiff
gmt grdcut @earth_day_30s_p.tif -R0/10/0/10 -Gonetile.png=PNG

onetile

It is not supported to upload the Gtiff file.

Esteban82 commented 10 months ago

I even could make a map with correct annotations:

gmt grdcut @earth_day_30s_p.tif -R-60/-50/0/10 -Gonetile.png=PNG
gmt grdimage onetile.png -Baf -png test

test

PaulWessel commented 10 months ago

Good. I noticed two further points:

  1. No need to append =PNG or =GTiff, just having the right file extension png, tiff etc. is enough for us to set the right driver.
  2. The tiff with our without =GTiff was still a geotiff and identical
Esteban82 commented 10 months ago

This looks good. Do we have those gdal "tools" in GMT? 

# 1. Create two tiles
gmt grdcut @earth_day_30s_p.tif -R-70/-60/0/10 -G1.tiff
gmt grdcut @earth_day_30s_p.tif -R-60/-50/0/10 -G2.tiff

# 2, Merge files
gdalbuildvrt mosaic.vrt ?.tiff
gdal_translate -of GTiff -co "TILED=YES" mosaic.vrt mosaic.tiff

# 3. Test merge file
gmt grdimage mosaic.tiff -Baf -png test

test

joa-quim commented 10 months ago

Do we have those gdal "tools" in GMT?

On Windows yes. Others, depend on how they installed GDAL.

PaulWessel commented 10 months ago

My GDAL on macOS has them. Perhaps

  1. Split those big images into managable tiles as done with for grids. E.g., write _src_tilerimg.sh based on _srctiler.sh (and rename that one _src_tilergrid.sh).
  2. Given -R, determine which tiles we need (as for grids) but then call gdalbuildvrt and _gdaltranslate to build the combo mosaic.
  3. Plot that thing.
joa-quim commented 10 months ago

Own builds are not guaranteed to have the gdal tools. But we have grdgdal that works as gdal_teanslate.

PaulWessel commented 10 months ago

The installers have the GDAL tools we use so can easily add a few more if needed. As for building from source, we require GDAL I think.

joa-quim commented 10 months ago

We indicate gdal-dev not the package with the binaries.

anbj commented 10 months ago

We indicate gdal-dev not the package with the binaries.

But gdal-bin is listed in the optional packages.

Esteban82 commented 10 months ago

Here I made a more complex test leaving some empty places to see if gdal can merge the tiles. It did it well.

test

Full script

# 1. Create tiling
gmt grdcut @earth_day_30s_p.tif -R-70/-60/0/10 -G1.tiff #=PNG
gmt grdcut @earth_day_30s_p.tif -R-60/-50/0/10 -G2.tiff #=PNG
gmt grdcut @earth_day_30s_p.tif -R-70/-60/-20/-10 -G5.tiff #=PNG

# 2, Merge files
gdalbuildvrt mosaic.vrt ?.tiff
gdal_translate -of GTiff -co "TILED=YES" mosaic.vrt mosaic.tiff

# 3. Test merge
gmt grdimage mosaic.tiff -Baf -png test