ufs forecast takes long time

[hailingz]

Thanks to @climbfuji for his earlier bugfix, we are able to run a 3dvar-forecast cycle using ufs-bundle.

For ROMEX, we are running JEDI with the full resolution background (c768) and c384 staticB. Currently the c384 staticB is provided on the layout=6x6 config. Also the current ufs-3var-fc workflow sets up the same layouts (processors) for both JEDI and ufs forecast. With layout=6 (216 processors total), each forecast time step takes ~317 second, and a 6h forecast takes 12.68h given the integration time is 150s in the ufs configuration (144 time steps in 6h).

Another data point is that with layout=10 (600 processors), each forecast time takes ~120 second, and a 6h forecast takes ~4.9h. This test was made available using a identical B.

Our question is that: if we should apply a different layout for DA and ufs forecast in the workflow to accelerate the forecast time?

[huishao-r]

@climbfuji @cmgas Could you please let us know whether such a change would be possible? Or anybody could give some guidance on making such a change?

[cmgas]

@huishao-r you could change it in the forecastUFS task: https://github.com/JCSDA-internal/skylab/blob/develop/models/gfs/tasks/forecastUFS.py#L35 and add lines to change the jedi_nodes, jedi_tasks_per_node and mpi_layout.

[climbfuji]

Closing this, instructions were provided that could speed up the model runs