WIP: Rename regression_test.py to use pytest

rjleveque commented 1 year ago

Also updated regression data for 2 examples with minor changes in results.

See also

rjleveque commented 2 months ago

Two tests are failing, tests/acoustics_2d_radial and tests/acoustics_2d_adjoint. The former ran fine on my laptop and the latter did not, but I updated the regression_data in commit e1ff225, and now it runs fine on my laptop.

@ketch or @mandli, could you try running pytest with this branch and see if how they behave for you?

mandli commented 2 months ago

I have the following with the full pytest output attached below.

acoustics_2d_radial

E           AssertionError:
E           Not equal to tolerance rtol=1e-14, atol=1e-08
E
E           Mismatched elements: 18 / 44 (40.9%)
E           Max absolute difference: 0.0021096
E           Max relative difference: 0.02010603

acoustics_2d_adjoint

E           AssertionError:
E           Not equal to tolerance rtol=1e-14, atol=1e-08
E
E           Mismatched elements: 3 / 51 (5.88%)
E           Max absolute difference: 0.00021523
E           Max relative difference: 0.00756248
E           Gauge Match Failed for gauge = 1
E             failures in fields: 0

pytest_output.txt

mandli commented 2 months ago

I noticed that some of the tests were behaving oddly and made some adjustments to the adjoint tests. Unfortunately, while I think it fixed one problem, it caused another for the 1D version to surface.

rjleveque commented 2 months ago

I tried running it on a linux machine and I got the same results as @mandli, and digging into the tests/acoustics_2d_radial results, the level 3 grids it makes are slightly different larger and hence the results start to diverge near the patch boundary. I thought it might be because I was using the ifort compiler, but redoing it with gfortran on linux didn't change the results (still different than with gfortran on my MacBook). In both tests I used a fresh clone of the repos plus this PR, so I think all the code should have been identical.

Maybe there's a cell with a value that's within rounding error of the flagging tolerance and if it gets flagged another row of cells is required in the patch.

But strangely it's not just in one patch but in all 5 of the level 3 grids in the figure below. In one computation they all start at y = -0.26 and in the other at y = -0.24.

@mjberger is this something worth pursuing, or should we just change the tolerance a bit for example and hope it goes away?

frame1

mjberger commented 2 months ago

it's hugely annoying, sometimes impossible, to track down this kind of thing. How soon does it diverge into the calculation? Is it the same for enough time to be a meaningful test if you shorten the run time?

— Marsha

On Jun 19, 2024, at 11:08 PM, Randall J. LeVeque @.***> wrote:

I tried running it on a linux machine and I got the same results as @mandli https://github.com/mandli, and digging into the tests/acoustics_2d_radial results, the level 3 grids it makes are slightly different larger and hence the results start to diverge near the patch boundary. I thought it might be because I was using the ifort compiler, but redoing it with gfortran on linux didn't change the results (still different than with gfortran on my MacBook). In both tests I used a fresh clone of the repos plus this PR, so I think all the code should have been identical.

Maybe there's a cell with a value that's within rounding error of the flagging tolerance and if it gets flagged another row of cells is required in the patch.

But strangely it's not just in one patch but in all 5 of the level 3 grids in the figure below. In one computation they all start at y = -0.26 and in the other at y = -0.24.

@mjberger https://github.com/mjberger is this something worth pursuing, or should we just change the tolerance a bit for example and hope it goes away?

frame1.png (view on web) https://github.com/clawpack/amrclaw/assets/720122/e2d0541e-0e44-4893-a103-60245a6ba26c — Reply to this email directly, view it on GitHub https://github.com/clawpack/amrclaw/pull/279#issuecomment-2179726045, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGUGC6VM4X7YHUJ6RJMGLDZIJBSJAVCNFSM6AAAAABJEYUUPSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZZG4ZDMMBUGU. You are receiving this because you were mentioned.

rjleveque commented 2 months ago

It's a small unit test problem and looking at fort.amr after setting verbosity and verbosity_regrid to 3, I see that the number of cells flagged already differs at time 0, on both levels 2 and 3. So it seems like it must be something dumb I'm missing...

clawpack / amrclaw

WIP: Rename regression_test.py to use pytest #279