Reduce some map data

[goord]

For the Xynthia flood and the locust stories we should try to reduce the wms map data as much as we can. Try to use short ints for the flood maps (cm) and floats for the global crop yield maps.

[sjvrijn]

I've started with the biggest file I could find: /data/receipt/thredds-docker/data/story_006/west_france_depth.nc (>1GB).

Because it uses NaN to keep land transparant, using int loses that information. (it would also only reduce the size by half to 520 MB). However, using a 4x4 coarsen the size reduces to 65 MB, and could probably be reduced further if needed. In general, is there a target size we're aiming for?

Files and python scripts used for size reductions have been added on the server.

west_france_depth_i16.nc :

west_france_depth_coarse4x4.nc:

[goord]

Hi @sjvrijn reducing spatial resolution is bit of a last resort option imo, but if all else fails that's what we should do. We can also take out some time steps.

If we convert to cm or mm, then cast to ints and finally apply aggressive compression, we could avoid coarsening.

It's difficult to gauge the desired size, but let's go for <100mb first and see how the server handles it.

[sjvrijn]

• story1/DS_cli_us.nc: saved with zlib=True, complevel=1. Size reduction 20 MB -> 5 MB
• story_005/crop_production_glob.nc: float64 -> float32, saved with zlib=True, complevel=1. Size reduction 241 MB -> 8.6 MB
• story_006/west_france_depth.nc: 3x3 coarsen, saved with zlib=True, complevel=1. Size reduction 1.1 GB -> 41 MB
• story_006/FRA_SSP{1,3,5}_rel_change_2020_2100.nc: float64 -> float32, saved with zlib=True, complevel=1. Size reduction 16 MB -> 1.4 MB (each)

All other remaining files are smaller than 10 MB

[goord]

Impressive achievements. I will test coming days whether we see better performance of the visualizer too.

[sjvrijn]

Performance is good, issue closed.