search

acalcutt / jaxa_AW3D30_to_MBTiles

Scripts to create a mbtiles file from jaxa AW3D30 elevation images. To download jaxa data please register an account with them first.
MIT License
17 stars 5 forks source link

Better compression #7

Open utagawal opened 8 months ago

utagawal commented 8 months ago

According to https://makina-corpus.com/sig-webmapping/optimisation-tuiles-mnt-rgb-ombrage-dynamique-mapbox-gl-maplibre-gl it could be possible to save up to 40% size by using Webp format.

They also have a way to reduce tha altitude precision by zoom level and would generate up to 80% space savings.

See implementation here : https://www.data.gouv.fr/fr/datasets/r/7ce01e48-fb89-4440-8098-78d7f758829a

Any chance that could be tested / implemented if this would be the case ?

acalcutt commented 8 months ago

Based on my recent testing in this thread https://github.com/ConservationMetrics/mapgl-tile-renderer/pull/44#issue-2143001167 , I would say the size difference is probably true. In the rendered tiles there webp was much smaller. The only thing that concerns me is webp compatibility. Now that most browsers are based on chrome and IE is unsupported, its probably not as much of and issue, but I feel like even firefox support is pretty recent ( though looking it seems most bowsers after 2020 support it now )

As for the adjusting precision, my last attempt of that I did not like. It worked ok for hillshade, but I wasn't happy at all with how it looked for terrain. It is possible their process looks better, but I'm not sure I have the time to work on it right now.

utagawal commented 8 months ago

Thanks. When you'll have the bandwidth : The second link gives the "for loop" to change the precision.

Le dimanche 3 mars 2024 à 18:06:44 UTC+1, Andrew Calcutt ***@***.***> a écrit :

Based on my recent testing in this thread ConservationMetrics/mapgl-tile-renderer#44 (comment) , I would say the size difference is probably true. In the rendered tiles there webp was much smaller. The only thing that concerns me is webp compatibility. Now that most browsers are based on chrome and IE is unsupported, its probably not as much of and issue, but I feel like even firefox support is pretty recent ( though looking it seems most bowsers after 2020 support it now )

As for the adjusting precision, my last attempt of that I did not like. It worked ok for hillshade, but I wasn't happy at all with how it looked for terrain. It is possible their process looks better, but I'm not sure I have the time to work on it right now.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

acalcutt commented 8 months ago

I looked at that, but it is very similar to what i tried before that did not work well for terrain https://github.com/acalcutt/jaxa_AW3D30_to_MBTiles/blob/smaller_terrainrgb/create_terrainrgb.sh

acalcutt commented 8 months ago

Since I have the JAXA geotiffs downloaded already and I've been wanting to test the compact feature I added to https://github.com/acalcutt/rio-rgbify/tree/compact , I made a run with webp z0-Z12 that you can download by torrent here https://techidiots.net/downloads/open-base-map-torrents/mbtiles/jaxa_terrainrgb_z0-12_webp-torrent/view

I am making a new png version right now so I can compare the actual space saving, since the old file I made was not deduplicated, which should help a lot for oceans where the jaxa dataset has no data

utagawal commented 8 months ago

Great news @acalcutt this is an impressive size improvement !

That could then revise the feasibility of https://github.com/acalcutt/jaxa_AW3D30_to_MBTiles/issues/3#issuecomment-1721678525 ?

acalcutt commented 8 months ago

I finished making the new png version of z0-Z12 https://www.techidiots.net/downloads/open-base-map-torrents/mbtiles/jaxa_terrainrgb0-12-torrent/view

The new file made with my compact feature looks like it only saved about 23GB, since the new file was 404GB vs the old png one I made that was 427GB. The WEBP one with the compact feature is only 263GB

utagawal commented 8 months ago

New Webp terrain layer is in production on our map server. Thanks to you we saved a lot of disc space and bandwidth. Have a look at https://maps.UtagawaVTT.com

acalcutt commented 8 months ago

Your map is looking great :-)

utagawal commented 5 months ago

In case you want to try another way (?) have a look here to see how they implement better compression : https://github.com/SilvioDallAcqua/terrain-rgb/tree/main?tab=readme-ov-file

using this command :

rio rgbify -b -10000 -i 0.1 --max-z 5 --min-z 5 --format webp -j 16 --round-digits 11 swissaltiregio_3857.tif swissaltiregio_05.mbtiles
rio rgbify -b -10000 -i 0.1 --max-z 6 --min-z 6 --format webp -j 16 --round-digits 10 swissaltiregio_3857.tif swissaltiregio_06.mbtiles
rio rgbify -b -10000 -i 0.1 --max-z 7 --min-z 7 --format webp -j 16 --round-digits 9 swissaltiregio_3857.tif swissaltiregio_07.mbtiles
rio rgbify -b -10000 -i 0.1 --max-z 8 --min-z 8 --format webp -j 16 --round-digits 8 swissaltiregio_3857.tif swissaltiregio_08.mbtiles
rio rgbify -b -10000 -i 0.1 --max-z 9 --min-z 9 --format webp -j 16 --round-digits 7 swissaltiregio_3857.tif swissaltiregio_09.mbtiles
rio rgbify -b -10000 -i 0.1 --max-z 10 --min-z 10 --format webp -j 16 --round-digits 6 swissaltiregio_3857.tif swissaltiregio_10.mbtiles
rio rgbify -b -10000 -i 0.1 --max-z 11 --min-z 11 --format webp -j 16 --round-digits 5 swissaltiregio_3857.tif swissaltiregio_11.mbtiles
rio rgbify -b -10000 -i 0.1 --max-z 12 --min-z 12 --format webp -j 16 --round-digits 4 swissaltiregio_3857.tif swissaltiregio_12.mbtiles
rio rgbify -b -10000 -i 0.1 --max-z 13 --min-z 13 --format webp -j 16 --round-digits 3 swissaltiregio_3857.tif swissaltiregio_13.mbtiles