GOSI10: NEMO optimisation and porting

[oceanaguiar]

• Completion of tiling and halo exchanges
• Adaptation for GPUs (following on work by Simon Mueller and STFC – Andy Porter).
• Incorporate some optimisations in GOSI configurations: for CPUs + RK3 in NEMOv5 (~first release in Jul 2024~)
• NEMOv5 official release planned for Sep 2024 but we are testing NEMOv5 beta release in our current GOSI10 prototype
• Mixed precision available in the next ~2 yrs (OFR&D requirement)
• Port GOSI configurations to new Met Office machines (GOSI9 already ported)
• [Testing XIOS3 with GOSI9]

[dcalve]

Delivery estimates

• Completion of tiling and halo exchanges
  • End of January 2024 (RK3 tiling and draft developer documentation)
  • End of June 2024 (developer documentation in final form)
• Adaptation for GPUs (following on work by Simon Mueller and STFC – Andy Porter)
  • End of February 2024 (single GPU performance in GO8, GOSI9, GOSI10)
  • End of March 2024 (multi-GPU scaling in GO8, GOSI9, GOSI10)
• Incorporate some optimisations in GOSI configurations: for CPUs + RK3 in NEMOv5 (first release in Jul 2024)
  • End of July 2024
  • This should just be a case of turning on the RK3 and tiling functionality, although I will likely need to change some bespoke GOSI code to work with the tiling (e.g. the vertically-interpolated MLD)
• Mixed precision available in the next ~2 yrs (OFR&D requirement)
  • ???
  • My work on this has generally been low priority
• Port GOSI configurations to new Met Office machines (GOSI9 already ported)
  • ???
• Testing XIOS3 with GOSI9
  • End of February 2024
  • Largely done, but work currently on hold & need to rerun performance experiments

[oceanaguiar]

I've added Tim Graham here (action after the last JMMP-GO meeting) for consultation on the time-step to adopt for NEMO, ensuring we use the same value for forced and coupled model runs.

[dcalve]

Potential issues from the NEMO 5 port

These are a list of potential issues from the NEMO 5 beta release notes identified by Diego and myself.

• New features

  • RK3 doesn't yet work with trends diagnostics (output by CMIP6 diagnostics), W&D (CO10) or data assimilation (Mike looking into this)

    Missing functionalities: RK3 cannot yet handle the trends, explicit time stepping, wetting and drying, and assimilation. Additional options like Shuman averaging for internal wave damping (ln_shuman=T) and the automation of the temporal dissipation for barotropic equations (nn_bt_flt=3) are available but not fully tested.

  We should use the Modified Leap Frog scheme for the GOSI10 NEMO 5 release, but we can still test RK3

  • New CPP keys have been added that are tied to the choice of vertical coordinate

    Reduction of the memory footprint. For instance, the type of vertical coordinates is controlled by cpp keys (key_vco_1d, key_vco_1d3d, key_vco_3d) to reduce memory access.

    • key_vco_1d = flat bottom (only works for z-coordinates)
    • key_vco_1d3d = 1D scale factor at w-level and 3D scale factor at t-level (used in most configurations).
    • key_vco_3d = full 3D scale factors (required for s-coordinates)

  For partial-step z-coordinates (zps) we should use key_vco_1d3d, but for more generalised vertical coordinates (e.g. multi-envelope or vanishing quasi-sigma) we must use key_vco_3d.

  • It isn't clear what the below change means for us- we use both these options in eORCA025

    The Courant number dependent implicit vertical advection (ln_zad_Aimp) is implemented for vector form in addition to the flux form.

  What approach was taken before if this is now supported? Requires testing/checking

• Removed features

  • VVL

  We should test the QCO vertical coordinate (key_qco) at v4.2.2 before porting to NEMO 5, as VVL will no longer be available

• Namelist changes

  • NEMO

  • nn_fct_imp has been added- this determines the method used for the adaptive-implicit vertical advection calculation in the FCT scheme

    In FCT, the implicit part can be approximated (nn_fct_imp=1) or fully accurate but costly (nn_fct_imp=2). Implicit is not yet coded for UBS tracer advection (ln_traadv_ubs).

    The default (1) is the new "optimised" method that is more computationally efficient. I think 2 corresponds to the old scheme. Should we worry about switching back to the old scheme for traceability, given that NEMO 5 will change results anyway?

  • ln_bt_av has been removed

    ln_bt_av is removed (replaced by the choice of nn_bt_flt)

    ln_bt_av=.true. is equivalent to nn_bt_flt=1- we already use this, so we just need to remove ln_bt_av from the suite namelist

  • SI3

  • rn_sinew has been added and needs checking

    If using the old parameterization (nn_icesal=2), be careful to change rn_sinew as well.

  • General points

  • Do we need to ensure that all namelist parameters appear in the suite rose-app.conf?

    • Benefit: easier to check what we are using, no surprises when the reference namelist changes (e.g. the nn_vvl_interp changes in NEMO 4.2.2)
    • Downside: we have to update the suite namelist more often

• Other code changes

  • Many arrays have been reduced in size

    Removal of unnecessary halo calculations

  We should take care to check that the GOSI code is consistent with these new array sizes. This is easy to check with the appropriate compiler options.

  • Tiling is now available with NEMO 5

  Tiling is off by default (ln_tile=.false.) and I suggest we keep it that way for now. The main reason for this is that we would also have to provide optimal tile sizes for each configuration and each machine that will be running GOSI10. Additionally, we wouldn't have to modify the GOSI code for the tiling.

[atb299]

@dcalve , @oceandie , thanks for this breakdown, it is really helpful! Regarding:

• Do we need to ensure that all namelist parameters appear in the suite rose-app.conf?

My personal preference would be to make use of the namelist_ref and namelist_cfg, with only changes relevant to a configuration in the namelist_cfg. There have been other 'gotchas' in the past where the default behaviour has changed (e.g. with the EOS back in v3.6, or the way ln_closea is handled in v4 > v4.2), so keeping conscious choices separate from unconscious choices can be helpful. That said, I don't work a lot with rose suites, and there are utilities around that can cleanly separate just the _cfg changes from a rose generated namelist.

[dcalve]

See #22

[dcalve]

Code changes when moving from VVL to QCO (key_qco)

Overview

When key_qco is used, various arrays associated with the vertical coordinate system are substituted for arithmetic expressions via CPP macros defined in src/OCE/DOM/domzgr_substitute.h90. For example:

#   define  e3t(i,j,k,t)   (e3t_0(i,j,k)*(1._wp+r3t(i,j,t)*tmask(i,j,k)))
#   define  e3u(i,j,k,t)   (e3u_0(i,j,k)*(1._wp+r3u(i,j,t)*umask(i,j,k)))
#   define  e3v(i,j,k,t)   (e3v_0(i,j,k)*(1._wp+r3v(i,j,t)*vmask(i,j,k)))
#   define  e3f(i,j,k)     (e3f_0(i,j,k)*(1._wp+r3f(i,j)*fmask(i,j,k)))
#   define  e3w(i,j,k,t)   (e3w_0(i,j,k)*(1._wp+r3t(i,j,t)))
#   define  e3uw(i,j,k,t)  (e3uw_0(i,j,k)*(1._wp+r3u(i,j,t)))
#   define  e3vw(i,j,k,t)  (e3vw_0(i,j,k)*(1._wp+r3v(i,j,t)))

These were introduced in NEMO 4.2 as a replacement for VVL and are now the only option in NEMO 5.0.

The macros are a bit more complicated in NEMO 5.0 due to the introduction of key_vco* CPP keys. E.g. e3t in the above example becomes:

# if defined key_vco_1d
#   define  e3t(i,j,k,t)   (e3t_1d(k)*(1._wp+r3t(i,j,t)*tmask(i,j,k)))
# elif defined key_vco_1d3d || defined key_vco_3d
#   define  e3t(i,j,k,t)   (e3t_3d(i,j,k)*(1._wp+r3t(i,j,t)*tmask(i,j,k)))
# endif

In summary, QCO causes several vertical coordinate arrays in the NEMO source code to be replaced by algebraic expressions during the preprocessing step of compilation. The main purpose of this is to reduce memory consumption- in the above examples, e3t is no longer a 4D variable but instead comprised of 2D and 3D variables. A very small impact on results should be expected.

More information can be found in the migration section of the NEMO user guide and in this ticket.

Implications

These array changes should generally have little impact, but some use cases to bear in mind are:

• The affected arrays cannot be assigned to
  • This is because they are now an arithmetic expression, which can't appear on the LHS of an assignment
• The affected arrays cannot be accessed via colon notation (i.e. e3t(:,:,:,:))
  • This is because this notation would be equivalent to e.g. (e3t_1d(:)*(1._wp+r3t(:,:,:)*tmask(:,:,:))), which would cause a conformance error due to the different shapes of the arrays
  • Therefore, the affected arrays must be accessed in a scalar fashion, i.e. via DO loops or specific indices, or copied to a working array
• The affected arrays cannot be passed as subroutine arguments
  • This is for the same reason that they cannot be accessed via colon notation
  • The solution for this specific use case is usually to copy the data to a working array (e.g. as done for e3t in diawri.F90) and use this as the argument

These will generally cause compilation failures, and so should be easy to spot.

[oceandie]

Thanks @dcalve, very clear! One thing - do you know by any chance if we have say on how to document / explain things/changes in the manual and/or official web pages? For example, I think the way the key_qco is explained in relation to the type of vertical coordinates available in NEMO in migration section is not correct and misleading for the users ...

[dcalve]

The manual and user guide have their own repositories:

• Manual
• User guide

We are definitely encouraged to contribute to these and any changes go through a review process like the NEMO code. New versions are generally published with each NEMO release, so any corrections or clarifications you would like to make for the NEMO 5 release would be very much appreciated!