search

MPAS-Dev / MPAS

Repository for private MPAS development prior to the MPAS v6.0 release.
Other
4 stars 0 forks source link

Adds LIGHT super-cycling capability #1484

Closed pwolfram closed 6 years ago

pwolfram commented 6 years ago

This merge provides the capability to do LIGHT super-cycling.

An example use case is

namelist changes

config_dt = '00:10:00'
config_AM_lagrPartTrack_compute_interval = '0000_01:00:00'

example particle file changes:

dtParticle = 3600 # time step for particles in s
timeIntegration = 4 #use RK4
pwolfram commented 6 years ago

Testing

A spun-up 32km SOMA test case was used with two cases over 30 days of simulation:

  1. 300s particle timestep at 600s compute time step (sub-cycled) using RK2
  2. 3600s particle timestep at 1 hr compute interval (super-cycled) using RK4

The error is greatest near regions of strongly varying flow, e.g., in initial regions near the shelf as well as in the bifurcation points of the jet; in general error is reasonably small and Results are reasonable and the computational step speedup is approximately a factor of 6 for the compute step.

Speedup for G RRS18to6 (on grizzly)

This provides a drastic speedup in the compute step of LIGHT, e.g., for speed up of a factor of 6 relative to use of RK2 with dtParticle=300, as previously used. Results are consistent with expected cost corresponding to computation frequency, time step frequency, and choice of Runge Kutta method.

Speed up can be tuned relative to accuracy via choice of dtParticle, timeIntegration, and config_am_lagrparttrack_compute_interval.

This capability will drastically reduce the computational cost of LIGHT's compute step in the 18to6, as previous literature suggests a 6 hr time step with a three-day computational frequency using RK4 would provide roughly a factor of 120X speedup relative to computing particles at the native 6-min timestep frequencies.

pwolfram commented 6 years ago

@vanroekel, files from this branch will be needed for application of LIGHT to the 18to6.

pwolfram commented 6 years ago

I've confirmed that the LIGHT and nightly test suites pass relative to ocean/develop.

mark-petersen commented 6 years ago

Passes nightly regression suite on grizzly with gnu/opt and intel/debug. All bfb with previous commit, as expected.

@pwolfram Does this need any corresponding changes in E3SM, such as any change in namelist flag settings?

pwolfram commented 6 years ago

@mark-petersen, we will need to change namelist and streams files for E3SM once we have the performance fully sorted out. I'm currently testing an approach to reduce restart cost. I don't think that effort should hold up this PR.

vanroekel commented 6 years ago

@mark-petersen and @pwolfram do the potential name list changes imply that LIGHT will be on by default in E3SM runs? If so, I think this requires a much broader discussion than here. I’m pretty uncomfortable with LIGHT on by default for E3SM.

pwolfram commented 6 years ago

@vanroekel, I think there quite a few challenges that would need surmounted before it could be on by default. I'm also not comfortable myself with this notion until I can demonstrate enhanced performance for IO and then there are other software engineering tasks that would also likely need to be completed first too. I'm afraid we are on a tangent here that is not pertinent to this PR.

mark-petersen commented 6 years ago

I get it. The take-home message is that we can merge this PR and not have any repercussions in E3SM for standard configurations. Correct?

pwolfram commented 6 years ago

Correct

From: Mark Petersen notifications@github.com Sent: Wednesday, January 10, 2018 5:37:31 PM To: MPAS-Dev/MPAS Cc: Phillip J. Wolfram; Mention Subject: Re: [MPAS-Dev/MPAS] Adds LIGHT super-cycling capability (#1484)

I get it. The take-home message is that we can merge this PR and not have any repercussions in E3SM for standard configurations. Correct?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/MPAS-Dev/MPAS/pull/1484#issuecomment-356785661, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEGMrZ2tPWnsZ2bUd6Fl7t4fxE45a8aoks5tJVfLgaJpZM4RYX3-.

mark-petersen commented 6 years ago

@vanroekel mentioned that this PR works on grizzly but had a problem on other machines. @pwolfram, I'd like to postpone this merge until we can confirm that it works on machines like theta and titan.

mark-petersen commented 6 years ago

This will be included in the next batch of small fixes for MPAS in E3SM, most likely on Feb 8.