search

CliMA / ClimaAtmos.jl

ClimaAtmos.jl is a library for building atmospheric circulation models that is designed from the outset to leverage data assimilation and machine learning tools. We welcome contributions!
Apache License 2.0
84 stars 17 forks source link

Flame tests are too brittle #2681

Open charleskawczynski opened 9 months ago

charleskawczynski commented 9 months ago

It seems that very minor changes can result in our flame allocation tests to break. I suspect that this is due to inference failures, and we should probably fix this. A quick patch for now, to at least alleviate some of the headache is to use a buffer, like we do for the threaded tests (which are inherently flakey).

Thoughts @Sbozzolo?

charleskawczynski commented 9 months ago

I think they're even flakey, which is worse. I updated the allocation limits in https://github.com/CliMA/ClimaAtmos.jl/pull/2673, and re-running still resulted in failure. This is another indicator that the issue is related to inference.

charleskawczynski commented 9 months ago

2673 is going to change buffer = occursin("threaded", job_id) ? 1.4 : 1 to buffer = occursin("threaded", job_id) ? 1.4 : 1.1 so that we have a bit of a buffer.

charleskawczynski commented 9 months ago

But we should revert this and fix the issue.

Sbozzolo commented 9 months ago

I had precisely the same thought and I would have proposed the same solution.