Chunking of sstaars netcdf file causing memory issues for ncwms tile rendering in prod

[jonescc]

See https://github.com/aodn/issues/issues/338

[ggalibert]

The chunking has been set so that it represents the whole geographic coverage of the product for one time step:

TEMP: type NC_SHORT, 3 dimensions, 7 attributes, compressed? yes, chunked? yes, packed? yes
TEMP compression (Lempel-Ziv without shuffling) level = 1
TEMP size (RAM) = 1*4500*6000*sizeof(NC_SHORT) = 27000000*2 = 54000000 bytes
TEMP dimension 0: DAY_OF_YEAR, size = 1 NC_SHORT, chunksize = 1 (Coordinate is DAY_OF_YEAR)
TEMP dimension 1: LATITUDE, size = 4500 NC_FLOAT, chunksize = 4500 (Coordinate is LATITUDE)
TEMP dimension 2: LONGITUDE, size = 6000 NC_FLOAT, chunksize = 6000 (Coordinate is LONGITUDE)

Since we would want to display the whole geographic coverage for any time step, I didn't think it would make any difference to set the chunking to something smaller than the whole geographic coverage.

What would you suggest?

[jonescc]

We have openlayers set to break layers up into tiles to improve performance. Each tile will be for a specific geographic region on the map (bbox). The bbox will depend on the zoom level. Tiles are rendered concurrently by the thredds instances we have available.

If the chunk contains the whole geographic coverage then the rendering of each tile requires the whole geographic extent to be loaded into memory - hence multiplying out the required memory by the number of tiles being rendered.

Some possible ways to address this:

• chunk lat/lon such that the max size of any chunk won't exceed a reasonable amount of memory (e.g. 5mb) and lat/lon chunks are square or close to square
• don't chunk so that the required data can be read directly from the disk
• create a lower resolution wms file

[ggalibert]

I have re-generated the dataset with 25 chunks (it should keep the size of a chunk at around 4MB) and uploaded it to S3. It works much better in prod already.