Add ability to restore salinity over longer time

[vanroekel]

This PR adds the ability to compute a salinity restoring tendency at a specified interval. The salinity restoring term is computed such that there is zero global mean. Thus if the salinity restoring term is computed once per day and applied with that same value every timestep, the global mean salinity will not change.

[vanroekel]

Restoring once per day instead of every timestep decreases the timer for salinity restoring from 76 seconds to 0.4 in a one month test run.

I will post images from restoring in ocean/develop and this PR soon.

[vanroekel]

@mark-petersen @jonbob and @maltrud This is a first attempt to only compute the salinity restoring tendency one time per day. I would appreciate it if you could all look.

[vanroekel]

@mark-petersen This PR should be ready. I've tested on the high res G-case on anvil. The salinity restoring diagnostic seems reasonable for computing restoring once per day and applying that tendency throughout the day. The salinity restoring was 12% of the runtime prior to this PR, and is now 0.06% of the run. The overall E3SM throughput increased from 0.52 SYPD to 0.68 SYPD post PR. (fyi @jonbob and @maltrud)

[vanroekel]

Output from a MPAS test EC60to30. One month long. This plot is the surface salinity averaged over the month pre PR The plot above is restoring tendency updated once per day. again surface salinity averaged over the month.

[mark-petersen]

Passes gnu and intel/debug MPAS-O nightly regression suite, but I think those don't have salinity restoring on. I put it in an ACME G case with salinity restoring, passes:

PET_Ln9.T62_oEC60to30v3.GMPAS-IAF.cori-knl_gnu
PEM_Ln9.T62_oEC60to30v3.GMPAS-IAF.cori-knl_gnu
ERS_Ld3.T62_oEC60to30v3.GMPAS-IAF.cori-knl_gnu

and these use salinity restoring.

[mark-petersen]

@vanroekel on the anvil G case you describe above, please check the average salinity in global stats. Before this PR, I think it stayed constant to all digits. After this PR, I expect that it would be non-constant over each day, but remain at the initial average long-term. We only write global stats daily, but you could check how close it stays to constant over your test run.

[vanroekel]

@mark-petersen the global average salinity for years 5 and 6 is below. The change to computing restoring once per day is about day 600. Also, it appears that the original salinityAvg is not constant to all digits (days 1-600). It seems that constant salinity to all digits is only possible if there is no drift to the sea surface height, which is not the case here (see second plot, which is ocean volume that appears to be inverted version of the salinity plot).

The change in this PR seems to have no discernible impact on the global mean salinity, which is what I expect. Recall that the routine that computes salinity restoring returns a surface flux that has zero global mean by construction. If we compute and apply the same tendency over the course of the day, there is no change to the global mean salinity.

@maltrud given the decrease in SSH should we tinker with the precipitation factor in this case?