search

CliMA / ClimaOcean.jl

🌎 Framework for realistic regional-to-global ocean simulations, and coupled ocean + sea-ice simulations based on Oceananigans and ClimaSeaIce. Basis for the ocean and sea-ice component of CliMA's Earth system model.
https://clima.github.io/ClimaOceanDocumentation/dev/
MIT License
29 stars 9 forks source link

Performance regression #132

Open simone-silvestri opened 2 months ago

simone-silvestri commented 2 months ago

A performance regression occurred between some commits of PR#104.

This commit ran in 4hr51 https://buildkite.com/clima/climaocean-examples/builds/103#_

while the latest one takes 5hr46 https://buildkite.com/clima/climaocean-examples/builds/200

Quite some change in performance. I think this is due to Oceananigans because there has been quite some development going on recently while PR #104 has not really changed so much.

I ll merge #104 but let's keep this in mind. I ll try to backtrack to see what happened.

glwagner commented 2 months ago

What changed in Oceananigans between those commits?

simone-silvestri commented 2 months ago

I am not sure, what comes into mind in terms of big PRs that might have affected the speed are this one https://github.com/CliMA/Oceananigans.jl/pull/3658 and this one https://github.com/CliMA/Oceananigans.jl/pull/3482 The former is supposed to have decreased the computational time so I am a bit puzzled unless among the last commits after the benchmarks something changed I have tested it in climaocean. The latter is only boundary conditions so I am not sure how it could have affected the speed. I ll do some tests

glwagner commented 2 months ago

What version of Oceananigans was performance measured on first, and what version is the second result from? Did you change compat between those?

simone-silvestri commented 2 months ago

Hmm, I tried benchmarks on Oceananigans and it looks like everything is in order. Also I have tried prototype_omip_simulation.jl in the prototype_omip_simulation folder on https://buildkite.com/clima/climaocean-examples/builds/103#_ forcing Oceananigans to main and indeed, the code is about 15% faster so it is a ClimaOcean thing.

The change happened between commit https://github.com/CliMA/ClimaOcean.jl/commit/826208d59b9ff106f0e067003d321971dc3f5612 and commit https://github.com/CliMA/ClimaOcean.jl/commit/ec6ddd2f6e0e6f71234586baf863365e49d6033d

but I only see the changes to the logarithmic profile which I don't think can have caused this difference. I ll prepare a MWE between the two commits