search

ecmwf-ifs / ectrans

Global spherical harmonics transforms library underpinning the IFS
Apache License 2.0
15 stars 30 forks source link

Add ability to inflate GPU buffers at runtime to accommodate postprocessing requirements #67

Closed samhatfield closed 2 months ago

samhatfield commented 4 months ago

By setting the ECTRANS_GPU_IF_PP_FACT environment variable, one can allocate more memory on the device to enable the bigger transforms required for IFS post-processing.

The variable should be an integer, which will be multiplied by NFLEV0 (number of levels on this task) and then added to other variables representing the standard complement of fields which are transformed on each timestep. This total will be used to determine the size of the "fields" dimension of transform work arrays.

In practice one needs to play around with this variable to find the lowest number that works. Typical values when using FULLPOS in the IFS are 3 or 4.

Enlarging the GPU device buffers will come with a performance penalty - every spectral transform will be more expensive, even the regular model transforms which aren't as big. However for now we have no choice, lacking a more intelligent solution (e.g. pool allocator). This is a good workaround for now.

If ECTRANS_GPU_IF_PP_FACT isn't set, nothing changes.

The branch name refers to JIRA issue DE-1127.

wdeconinck commented 2 months ago

Hi @samhatfield this looks good, but out of curiosity could you spell out what ECTRANS_GPU_IF_PP_FACT stands for? All of this should hopefully not be necessary anymore once we have a GPU memory pool infrastructure.

samhatfield commented 2 months ago

ECTRANS_GPU_IF_PP_FACT is not the best name now that I think about it. It's supposed to mean ecTrans GPU : multiplier of NFLEV0 (FACTor) to determine value of IF_PP - the number of postprocessing Fields. You could rename it ECTRANS_GPU_POSTPROCESSING_NFLEV0_MULTIPLIER or something.