PM tiles cloud job

[nmarchio]

Recipe: https://gist.github.com/stuartlynn/b210e5edb740169ccf70a7dbf10bde69

Dependency: https://github.com/felt/tippecanoe

[nmarchio]

Scratch notes: https://docs.google.com/document/d/1gWnYYviN8frB7jJQrmHTqpJ0WfxuthrSNq9TnYPI-hw/edit

[nmarchio]

Improved recipe: https://github.com/Matico-Platform/pmtiles-utils

See issues in repo for next steps

[nmarchio]

This issue involves:

• Deploy the the workflow outlined in the pmtiles-utils repo https://github.com/Matico-Platform/pmtiles-utils workflow in AWS job
• Upload to S3, trigger Glue, run VM (in FarGate or Batch). output to different S3 and return link
• Set up a VM with sufficient memory (64GB)
• Install dependencies the osgeo/gdal docker image (v3.5+) docker run -it osgeo/gdal:latest and https://github.com/felt/tippecanoe then install pmtiles-utils requirements
• Ingest 8.5GB GeoParquet (.parquet) from S3 here https://mnp2.s3.us-east-2.amazonaws.com/subsaharan_blocks.parquet
• Apply pmtiles-util functions
• Write the FlatGeoBuf (.fgb) to S3 bucket first (check if it exists, if it does exist skip the parquet to fgb conversion)
• Convert FlatGeoBuf to several different .mbtiles at different resolutions (develop preset recipes)
• Convert .mbtiles to pmtiles and write to final S3 destination

[nmarchio]

Data link: https://uchicago.box.com/s/7x1d4zg6xilbzpv0jdtan2cs9bpv08l2

Store data in root directory and run:

# Check if mali.fgb is present if it isn't run:
python3 convert --input=mali.gpkg --output=mali_z08-11.pmtiles --config=tiles_config_z08-11.yaml --retain=True
# retain=True keeps the intermediary files .fgb and .mbtiles
# Check if mali.fgb is present if it *is* run:
python3 convert --input=mali.fgb --output=mali_z08-11.pmtiles --config=tiles_config_z08-11.yaml --retain=False
python3 convert --input=mali.fgb --output=mali_z11-14.pmtiles --config=tiles_config_z11-14.yaml --retain=False

Converts mali.gpkg > mali.fgb > mali_z08-11.mbtiles > mali_z08-11.pmtiles

Feature enhancements to the pmtile-utils repo:

• Option to split out the .gpkg to .fgb conversion step (this is really just running ogr2ogr which is a GDAL function)
• Create a way to templatize the .yaml so we can sub in different zoom parameters (one idea: make a tiles_config_template.yaml where the maximum-zoom: __maxzoom__ the use sed in bash to replace maxzoom with an arbitrary number.

[nmarchio]

• test gpkg to fgb conversion using ogr2ogr to see its faster

[manmartgarc]

• test gpkg to fgb conversion using ogr2ogr to see its faster

Running with pyogrio took 33.05s while running with ogr2ogr took 19.36s.

This is for converting .gpkg files to .fgb.