Autoset nprocs and improve max_blocks estimate

[anton-seaice]

PR checklist

• [ ] Short (1 sentence) summary of your PR: Contributes to #945
  This improves setting nprocs and max_blocks automatically.

• [ ] Developer(s): @anton-seaice @minghangli-uni

• [ ] Suggest PR reviewers from list in the column to the right. @apcraig

• [ ] Please copy the PR test results link or provide a summary of testing completed below. Needs doing

• How much do the PR code changes differ from the unmodified code?

  • [ ] bit for bit
  • [ ] different at roundoff level
  • [ ] more substantial

• Does this PR create or have dependencies on Icepack or any other models?

  • [ ] Yes
  • [x] No

• Does this PR update the Icepack submodule? If so, the Icepack submodule must point to a hash on Icepack's main branch.

  • [ ] Yes
  • [x] No

• Does this PR add any new test cases?

  • [ ] Yes
  • [ ] No

• Is the documentation being updated? ("Documentation" includes information on the wiki or in the .rst files from doc/source/, which are used to create the online technical docs at https://readthedocs.org/projects/cice-consortium-cice/. A test build of the technical docs will be performed as part of the PR testing.)

  • [ ] Yes
  • [ ] No, does the documentation need to be updated at a later time?
    • [ ] Yes
    • [ ] No

• [x] Please document the changes in detail, including why the changes are made. This will become part of the PR commit log.

This change allows nprocs to be set to -1 in 'ice_in' and then the number of processors will be automatically detected.

This change improves the automatic calculation of max_blocks to give a better (but still not foolproof) estimate of max_blocks if it is not set in ice_in.

[apcraig]

This looks pretty good. A couple thoughts.

• Setting nprocs=get_num_procs does not happen all the time for the cpp CESMCOUPLED. I think we'll want backwards compatibility on that for now.
• The max_blocks estimate is improved, but it is still focused on even distribution of the 2d blocks. There are decompositions where it still will not work in general, decompositions that are more equal weighted in terms of estimated work vs number of blocks.

The modifications I'm working on handle the nprocs thing in much the same way, but also refactor the max_blocks calculation so each proc (mpi task) has a locally defined max_blocks that matches the decomposition exactly. I'm still developing and testing to make sure it'll work properly.

I'm happy to merge this as a first step (if the CESMCOUPLED thing is fixed). I could then update the implementation if I make additional progress. Thoughts?

[anton-seaice]

I'm happy to merge this as a first step (if the CESMCOUPLED thing is fixed). I could then update the implementation if I make additional progress. Thoughts?

Lets go with this. I think I made the related changes to do this.

I guess we should add some tests?

[apcraig]

I will run a full test suite once comments are addressed and the PR is no longer draft. Thanks!

[anton-seaice]

I will run a full test suite once comments are addressed and the PR is no longer draft. Thanks!

My question here is whether we should add some test cases for nprocs=-1. It may be a bit of work though, because most of the scripts are written around setting this explicitly.

[apcraig]

I will run a full test suite once comments are addressed and the PR is no longer draft. Thanks!

My question here is whether we should add some test cases for nprocs=-1. It may be a bit of work though, because most of the scripts are written around setting this explicitly.

Please run a test manually with nprocs=-1 to make sure it works. I think the question is maybe whether we should have nprocs=-1 be the default setup for all tests. You could try that and see if everything runs and is bit-for-bit. I think you'd need to set nprocs=-1 in configuration/scripts/ice_in and then remove "nprocs = ${task}" in cice.setup. Is that the direction we want to go?

[anton-seaice]

Our intermittent failure is back !

https://github.com/CICE-Consortium/CICE/actions/runs/9024916341/job/24799676376 failed https://github.com/ACCESS-NRI/CICE/actions/runs/9024916168/job/24799675998 passed

But they are the same commit

[apcraig]

I restarted the github action failure and it passed. Ugh....