Upload fixed salt_sfc_restore.nc file and develop input file sanity checks

[aekiss]

@russfiedler spotted a serious interpolation problem in the 0.1deg salt_sfc_restore.nc file: https://arccss.slack.com/archives/C6PP0GU9Y/p1511838112000214 (the offending file can be found here: /short/v45/aek156/access-om2/input/mom_01deg/salt_sfc_restore-BAD-DO-NOT-USE.nc)

This bad input/mom_01deg/salt_sfc_restore.nc file is present in at least the two latest input tarfiles /short/public/access-om2/input_c22b3335.tar.gz and /short/public/access-om2/input_154112e8.tar.gz so a user will get these faulty files when cloning https://github.com/OceansAus/access-om2.git followed by get_input_data.py.

There is a correct 0.1deg version (used by Fanghua) here: /short/x77/fw4078/mom/input/mom01/climatology/mean_salinity_of_0_and_10m.nc.

I've also checked other resolutions in the latest /short/public/access-om2/input_154112e8.tar.gz

• input/mom_1deg/salt_sfc_restore.nc and input/mom_1deg-old_grid/salt_sfc_restore.nc are OK
• input/mom_025deg/salt_sfc_restore.nc doesn't have the same fault as the 0.1 case, but it doesn't specify a FillValue for SALT so that needs fixing.

We need to

• [ ] upload the correct version /short/x77/fw4078/mom/input/mom01/climatology/mean_salinity_of_0_and_10m.nc (renamed as input/mom_01deg/salt_sfc_restore.nc) to a new tarfile in /short/public/access-om2/ and http://s3-ap-southeast-2.amazonaws.com/dp-drop/access-om2/ so it will be served up for new installs via get_input_data.py (don't use /g/data/ua8/MOM/salinity_restoring/01/mean_salinity_of_0_and_10m.nc, this has incorrect variable names). We could also correct the missing FillValue in the 0.25deg version input/mom_025deg/salt_sfc_restore.nc while we're at it. (@nicjhan is this something you'd normally handle?)
• [ ] make it hard/impossible to use the bad input tarfiles in /short/public/access-om2/ and http://s3-ap-southeast-2.amazonaws.com/dp-drop/access-om2/ in further experiments? (I mean at least /short/public/access-om2/input_c22b3335.tar.gz and /short/public/access-om2/input_154112e8.tar.gz, but probably all the rest also) -- we probably shouldn't delete them (so we keep a record of previous setups) but could turn off their read flags.
• [ ] write some test code to detect and prevent this problem in future (eg check that fields in non-land cells in restoring and initial condition files have values lying in acceptable ranges) - eg the nco commands ncwa -y min -v salt salt_sfc_restore.nc minsalt.nc; ncdump -v salt minsalt.nc should give a positive salt value.
• [ ] check whether this problem exists in any previous 0.1deg runs we've done

More generally, this points to a need for a more systematic, automated method to verify and maintain the integrity of our input files. Can something like that be done with the Jenkins test suite? I'm thinking a simple bounds check for all variables to confirm they don't have absurd values.

[ofa001]

@aekiss and @russfiedler I have a question and it may relate to these fields for 0.1 and 0.25 models and the negative/ low salinities and the new grids Russ built are the values right at the end of channels, or places like Baltic Hudson Bay. I was just remembering some Mk2/3 history (with Barrie's passing) and we had to increase our exchange coefficients across narrow straits to prevent negative salinities there (I think it was the first attempt at Mk3.0 as flux correction would have kept us on track in Mk2) I noticed you were increasing the size of your salinity restoring terms so may achieve the same result. But I do think you should look into the stability issues that the Met office found in the lower levels of CICE thermodynamics when the added an extra convergence criterion I mentioned earlier in the week it was again tripped in a low salinity region.

[aidanheerdegen]

I should have said, I have fixed the variable names for the smooth WOA sss restoring files in /g/data/ua8/MOM/salinity_restoring. Is there anything else that needs fixing in the metadata for those files?

md5sum /g/data1/ua8/MOM/salinity_restoring/*/salt_sfc_restore.nc
441cd4ed3336f5d8567dedf55b993160  /g/data1/ua8/MOM/salinity_restoring/01/salt_sfc_restore.nc
f6e39a106494e7a2f0232963cfb84220  /g/data1/ua8/MOM/salinity_restoring/025/salt_sfc_restore.nc
1737e61aafe7b2a09f25350d256106d0  /g/data1/ua8/MOM/salinity_restoring/10/salt_sfc_restore.nc

[aekiss]

thanks @aidanheerdegen /g/data1/ua8/MOM/salinity_restoring/10/salt_sfc_restore.nc has the wrong variable name for salinity and the path is confusing (should 10 be 1?) Others look OK.

[aekiss]

thanks @ofa001 it's been a busy week but I'll hopefully get a chance to look at that properly next week

[aidanheerdegen]

Grrrr. I was sure I had double-checked that. Thanks for picking it up

Shall we try that again then

md5sum /g/data1/ua8/MOM/salinity_restoring/*/salt_sfc_restore.nc
441cd4ed3336f5d8567dedf55b993160  /g/data1/ua8/MOM/salinity_restoring/01/salt_sfc_restore.nc
f6e39a106494e7a2f0232963cfb84220  /g/data1/ua8/MOM/salinity_restoring/025/salt_sfc_restore.nc
d95d45424492bad16d647c104e78b442  /g/data1/ua8/MOM/salinity_restoring/10/salt_sfc_restore.nc

I have used 10 = 1.0 to be consistent with 01 = 0.1. Makes sense in my brain.

[aekiss]

Yep, looks good, thanks @aidanheerdegen .

@nicjhan , @AndyHoggANU - should we replace the existing salt restoring files with these? I don't know to what extent they differ from the ones we've used for the 1deg and 025deg runs so far.

@aidanheerdegen - the salt field in /g/data1/ua8/MOM/salinity_restoring/01/salt_sfc_restore.nc is identical to @FanghuaWu's /short/x77/fw4078/mom/input/mom01/climatology/mean_salinity_of_0_and_10m.nc, right?

[aidanheerdegen]

@aekiss ncdiff shows no difference

[nichannah]

@aekiss, @aidanheerdegen do we have a conclusion? I've replaced the 0.1 deg but should we do the same for 0.25 and 1 deg?

[aidanheerdegen]

We should look at a masked version of the restoring files to see if there are any problematic values over ocean cells. If not then they shouldn't be replaced as there are long spin up runs of the 1 deg and 0.25 deg with the existing files.

We'll need @AndyHoggANU's opinion.

[AndyHoggANU]

Hi @nicjhan @aidanheerdegen @aekiss, I think we should replace these files. Yes, I know they will differ slightly from our current spinup, but there is no guarantee that we will ever find the perfect time to update these files. So, let's update the repository - if people want to run consistent runs, then the old input directory is still there for them. Sound reasonable?