search

JCSDA-internal / soca

Sea-ice Ocean Coupled Assimilation
Apache License 2.0
13 stars 4 forks source link

Speed up and revive a few soca_covariance_training ctests #782

Closed liuxiao37k closed 2 years ago

liuxiao37k commented 2 years ago

Description

A few soca_covariance_training ctests have been previously turned off due to slow execution.

Solution

Try to speed up these tests using coarsened grid and by reducing number of variables. Consolidate the tests if possible.

liuxiao37k commented 2 years ago

@guillaumevernieres Hmmm... So what are this and this? Maybe limit the variables to one to revive these tests?

@travissluka Sorry, I should have checked more carefully before creating the issue...

guillaumevernieres commented 2 years ago

@guillaumevernieres Hmmm... So what are this and this? Maybe limit the variables to one to revive these tests?

@travissluka Sorry, I should have checked more carefully before creating the issue...

@liuxiao37k the tests are commented out. But yes, I did forget they were part of the ci at one point ... Oops! I would suggest we turn them back on and speed them up. They could be done using the sub-sampled grid and just 1 variable.

liuxiao37k commented 2 years ago

@liuxiao37k the tests are commented out. But yes, I did forget they were part of the ci at one point ... Oops! I would suggest we turn them back on and speed them up. They could be done using the sub-sampled grid and just 1 variable.

Sounds good to me, if @travissluka doesn't vote against it. I'll provide info on execution time after thinning the grid and variables.

guillaumevernieres commented 2 years ago

FYI (just in case) @liuxiao37k , there is already a sub-sampled grid generated as part of the ctests (test 2 & 3). Use that geometry. If that bump test is done on a coarser grid and applied to only one variable, it might end up being OK timing wise.

travissluka commented 2 years ago

Sounds good, thanks for taking a look at it

liuxiao37k commented 2 years ago

I've had some fun and am ready to move on with the branch feature/bump_cov_fun.

To freshen our memory:

  1. there are a few bump cov training ctests added a while back, but turned off due to slowness.
  2. one option is to use the _small grid, but our _small grid isn't small enough, so we decided first to replace it with a smaller grid.

With this branch feature/bump_cov_fun, we revived a bump_cov_training ctest based on the new small grid. It requires a minimum of 4 ensemble members as an input to bump:. Several options have been explored:

  1. create and use an enspertlowres ctest to generate the pert.lowres.ens.*.an input files, which however requires populating /INPUT_small with the full mom6 initialization package, as well as forcing files if advance_mom6 is used to obtain PT6H files.
  2. create a ctest/application to ncks the pert.ens.*.ana (regular grid) files after they are generated by enspert ctest, which however also comes with a cost.
  3. pre-generate pert.lowres.ens.*.an files (~900k, 4 files) and upload them via git-lfs.

Changes in this branch reflects Option 3 above. To me it seems the cheapest option, but Options 1&2 make sense to me too.

Parameters used in parameters_bump_cov.yml is according to an older version from @guillaumevernieres. I was advised to keep them as they were.

The new parameters_bump_cov ctest takes about 5-6 seconds to complete. The results look reasonable, and figures are attached below.

Please comment @travissluka @guillaumevernieres @ShastriPaturi. Thank you!

image
guillaumevernieres commented 2 years ago

Sounds good to me @liuxiao37k , just create a PR when ready.