Input file validation?

[angus-g]

Maybe this isn't quite payu's domain, but a large number of very arcane errors show up in MOM6 because of input/forcing files with _FillValue set to NaN. Maybe as part of payu's manifest step, it could warn/error for suspect files. Of course, the model should be doing its own validation, but I think the root of these issues is in FMS so it's opaque and unlikely to be an easy avenue to pursue.

[aidanheerdegen]

Maybe as part of payu's manifest step, it could warn/error for suspect files

You want to check for incorrect attributes in the manifest step? For what purpose? To identify them and fix them?

[angus-g]

For what purpose? To identify them and fix them?

I'm not proposing that payu fix them, but because it's already looking at all the inputs, it saves continually hitting errors stemming from the same root of having an incorrect _FillValue (or missing modulo, etc.). Anyway, just getting a thought out.

[aidanheerdegen]

it saves continually hitting errors stemming from the same root of having an incorrect _FillValue (or missing modulo, etc.)

I'm a bit dense today, but I don't know the mechanism that would avoid these errors being flagged when used in the model short of fixing the files.

[angus-g]

Oh, it would definitely require manual intervention. But with a NaN _FillValue, FMS/MOM6 won't warn you at all. It'll plug away, and lead to hard-to-debug nonsense (from unphysical values, to seemingly-random segfaults). If this could generate a warning at the payu run stage, it could at least give a hint of where to look first.

[aidanheerdegen]

I think this probably a bit out of scope for payu. We were just thinking we might be able to get rid of the dependency on netCDF4 with #348

[aidanheerdegen]

Doesn't mean we couldn't make a utility to do this and call with the payu userscript functionality.

[angus-g]

Yeah, it does seem in the realm of feature creep. It'd probably be more valuable/flexible as an external utility.