reduce size of live data

[peterdudfield]

Detailed Description

Would be great to reduce the size of the NWP data. I think this could be done with a better compression.

Context

• Current szie is 1.2 GB - https://s3.console.aws.amazon.com/s3/buckets/nowcasting-nwp-development?region=eu-west-1&prefix=data/&showversions=false
• currently saves as .netcdf file
• other compressions has been discusse here - https://github.com/openclimatefix/ocf_datapipes/issues/132#issuecomment-1398195182

Possible Implementation

use Blosc Zstd compression

[jacobbieker]

I would try the new ocf_blosc2 Blosc2 ZSTD one, as that does give better results than the original Blosc Zstd

[peterdudfield]

Just so it super easy to do, do you a link to some code, where you save it using ocf_blosc2,

[jacobbieker]

Yeah, it would be something like this, which I use for the new satellite zarrs:

from ocf_blosc2.ocf_blosc2 import Blosc2

def write_to_zarr(dataset, zarr_name, mode, chunks):
    mode_extra_kwargs = {
        "a": {"append_dim": "time"},
        "w": {
            "encoding": {
                "data": {
                    "compressor": Blosc2("zstd", clevel=5),
                },
                "time": {"units": "nanoseconds since 1970-01-01"},
            }
        },
    }
    extra_kwargs = mode_extra_kwargs[mode]
    dataset.chunk(chunks).to_zarr(
        zarr_name, compute=True, **extra_kwargs, consolidated=True, mode=mode
    )

[devsjc]

Update deployed NWP consumer (to version 1.2.2):

• [x] dev via Terraform cloud run
• [ ] prod

[devsjc]

Updated forecaster to include OCF blosc 2 library (to version 1.3.11):

• [x] dev via Terraform cloud run
• [x] prod via Terraform cloud run

[peterdudfield]

great work @devsjc , what size did the NWP go down to using this method?

[devsjc]

Not quite there yet, I'll let you know!

[devsjc]

@peterdudfield which repository holds the national model? I will need to update that to be able to read the newly compressed NWP data as well.

[devsjc]

Seeing the following error in the forecaster in cloudwatch:

expected shape=(7, 24, 24, 11) actual shape (4, 24, 24, 11)

[devsjc]

Latest version does not seem to show the same error in cloudwatch

[devsjc]

Wrong shape error has occured again. @peterdudfield is this an expected error for the forecaster?

[devsjc]

NWP task is exiting with an out of memory error - must be the case that compression with Blosc2 takes more memory as that's the only change that has been implemented in that container. Increasing the memory: Dev: pr: https://github.com/openclimatefix/ocf-infrastructure/pull/251 tf: https://app.terraform.io/app/openclimatefix/workspaces/nowcasting_infrastructure_development-eu-west-1/runs/run-gkqyNt5GZj8vQpKX Prod:

[peterdudfield]

I just rolled back nwp to 1.2.0 from 1.2.2 on devlopment as it was causing an issue on development

https://eu-west-1.console.aws.amazon.com/cloudwatch/home?region=eu-west-1#logsV2:log-groups/log-group/$252Faws$252Fecs$252Fconsumer$252Fnwp$252F/log-events/streaming$252Fnwp-consumer$252F52032316e23c4ca5b31f8a4aa527cd4c