search

ACCESS-NRI / um2nc-standalone

Standalone um2nc/um2netcdf to unify development efforts in a single repository.
Apache License 2.0
0 stars 0 forks source link

Feature Requirement: conversion to 32 bit #7

Open truth-quark opened 3 months ago

truth-quark commented 3 months ago

From https://github.com/ACCESS-NRI/access-esm1.5-configs/issues/3:

TODO: what is the requirement here? TODO: covert all fields from 64 to 32 bit if option selected? TODO: what section of code does this correspond to?

TODO: side issue, could tie link this to adding deprecation warnings to NETCDF3, gently encouraging migration.

MartinDix commented 3 months ago

The ESM 1.5 UM files normally have 64 bit data. CMIP6 requires 32 bit output and converting to 32 bit as soon as possible in the workflow saves space. Therefore the default should be 32 bit output.

However when debugging it can be useful to get the full 64 bit output so this is an option. It's not normally necessary in the standard post-processing workflow.

truth-quark commented 2 weeks ago

As of commit 120dc47, 64 to 32 bit conversion is a standalone function:

https://github.com/ACCESS-NRI/um2nc-standalone/blob/120dc47d4a3c131c18c3f1030167ff29abcb37b3/umpost/um2netcdf.py#L936-L963

This only converts cube data fields to 32 bit, about 2/3rds of the way through the workflow in cubewrite() which is called by process():

https://github.com/ACCESS-NRI/um2nc-standalone/blob/120dc47d4a3c131c18c3f1030167ff29abcb37b3/umpost/um2netcdf.py#L322-L329

when debugging it can be useful to get the full 64 bit output so this is an option. It's not normally necessary in the standard post-processing workflow.

@MartinDix the conversion call can be moved earlier in the workflow, although it's a RAM vs disk space optimisation. I'm not sure if there are performance gains using 32 bit over 64 bit numpy arrays on 64 bit systems.

Do you have any known requirements to do the conversion earlier in the workflow?