search

ACCESS-NRI / access-om3-wav-configs

ACCESS-OM3 MOM6-CICE6-WW3 configurations
0 stars 1 forks source link

1deg jra55do iaf.cherrypicks #47

Closed minghangli-uni closed 3 weeks ago

minghangli-uni commented 3 weeks ago

Update PE layout and related cice blocks

minghangli-uni commented 3 weeks ago

How many cores are used for wav, 24?

dougiesquire commented 3 weeks ago

Good question. I just took a look at how a similar configuration is configured by CIME (on the cheyenne supercomputer):

Without WW3:

With WW3:

I think maybe just give it 24 for now and we can test this properly later.

minghangli-uni commented 3 weeks ago

Thanks @dougiesquire. I've made the change.

anton-seaice commented 3 weeks ago

Surely we want at least as many cores as the ocean ? i.e. 240

anton-seaice commented 3 weeks ago

Also, its possible there will be WW config files that need changing too.

minghangli-uni commented 3 weeks ago

Surely we want at least as many cores as the ocean ? i.e. 240

24 cores can be like a starting point. Can we update this later?

Also, its possible there will be WW config files that need changing too.

I am not very familiar with WW configuration. What config files might need changing?

dougiesquire commented 3 weeks ago

Surely we want at least as many cores as the ocean ? i.e. 240

Why do you say that? I had thought that the scaling was limited by the number of spectral bands (without using an unstructured mesh). How many bands are there?

dougiesquire commented 3 weeks ago

Also, its possible there will be WW config files that need changing too.

@ezhilsabareesh8?

ezhilsabareesh8 commented 3 weeks ago

Also, its possible there will be WW config files that need changing too.

There's no need of any changes in the WW3 config files for PE layout modifications. However it is worth checking the wav.log during the test run for any changes, I can do that.

dougiesquire commented 3 weeks ago

Thanks @ezhilsabareesh8

anton-seaice commented 3 weeks ago

I believe there are 15 spectral bands. From ad-hoc discussions I thought we were expecting it to be a similar resource need to the ocean model. Which is what the research from CIME shows ? ( i.e. they use 300 cores )

dougiesquire commented 3 weeks ago

Okay, I did some basic tests that show that 24 is definitely not a good choice - this runs slower than prior to these changes despite using 5x the number cores 😅.

I think this requires proper testing. Unless anyone wants to take that on now, I suggest we close this PR (and https://github.com/COSIMA/MOM6-CICE6-WW3/pull/46) and open new ones when we have scaling results to inform our decision.

minghangli-uni commented 3 weeks ago

this runs slower than prior to these changes despite using 5x the number cores

Thanks @dougiesquire. This is not a good choice though. I agree to close this #47 and #46 for now until we have proper scaling results. I will do a scaling test and then open an issue in this repo.