Closed francispoulin closed 2 years ago
After the big changes yesterday I decided to run the tests to make sure everything was working
Are you running branch used in PR #1174 ? The tests don't pass over there --- there's still some work to do. But we should discuss what needs to be done to get the tests to pass over on that PR (and then merge it into master
when the tests do pass).
I thought I was but I am using master so I guess not.
@francispoulin Hope you don't mind that I formatted your original post to put triple backticks (```) around the test results to make them easier to read.
If those tests failed on master, would be good to investigate why... Can you post the full output from running the tests?
Not sure why checkpointer would fail, but I could see some of the regression tests failing if your GPU is very different maybe. Do you know what GPU you used to run the tests with? Might be good for Oceananigans to print this info as part of runtests.jl
.
Thanks for cleaning it up @ali-ramadhan , and I will try and remember to do this in the future. Looks much better.
The output is pretty long, as you know, but I will copy the highlights below. If you do want to know all the details I can rerun it and probably figure out how to output the result to a file and then maybe include that.
Checkpointer [GPU]: Test Failed at /home/fpoulin/software/Oceananigans.jl/test/test_output_writers.jl:565
Expression: all(test_model.velocities.u.data .≈ true_model.velocities.u.data)
Stacktrace:
[1] macro expansion at /home/fpoulin/software/Oceananigans.jl/test/test_output_writers.jl:565 [inlined]
[2] macro expansion at /home/fpoulin/.julia/packages/GPUArrays/ZxsKE/src/host/indexing.jl:64 [inlined]
[3] test_model_equality(::IncompressibleModel{QuasiAdamsBashforth2TimeStepper{Float64,NamedTuple{(:u, :v, :w, :T, :S),Tuple{Field{Face,Cell,Cell,OffsetArray{Float64,3,CuArray{Float64,3}},Re
Checkpointer [GPU]: Test Failed at /home/fpoulin/software/Oceananigans.jl/test/test_output_writers.jl:565
Expression: all(test_model.velocities.u.data .≈ true_model.velocities.u.data)
Stacktrace:
[1] macro expansion at /home/fpoulin/software/Oceananigans.jl/test/test_output_writers.jl:565 [inlined]
[2] macro expansion at /home/fpoulin/.julia/packages/GPUArrays/ZxsKE/src/host/indexing.jl:64 [inlined]
[3] test_model_equality(::IncompressibleModel{QuasiAdamsBashforth2TimeStepper{Float64,NamedTuple{(:u, :v, :w, :T, :S),Tuple{Field{Face,Cell,Cell,OffsetArray{Float64,3,CuArray{Float64,3}},Re
Ocean large eddy simulation [GPU]: Test Failed at /home/fpoulin/software/Oceananigans.jl/test/regression_tests/ocean_large_eddy_simulation_regression_test.jl:122
Expression: all(test_fields.u .≈ correct_fields.u)
Stacktrace:
[1] run_ocean_large_eddy_simulation_regression_test(::GPU, ::VerstappenAnisotropicMinimumDissipation{Float64,Float64,Float64,Float64}) at /home/fpoulin/software/Oceananigans.jl/test/regression_tests/ocean_large_eddy_simulation_regression_test.jl:122
[2] macro expansion at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:77 [inlined]
[3] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
[4] macro expansion at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:74 [inlined]
[5] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
[6] top-level scope at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:60
Ocean large eddy simulation [GPU]: Test Failed at /home/fpoulin/software/Oceananigans.jl/test/regression_tests/ocean_large_eddy_simulation_regression_test.jl:123
Expression: all(test_fields.v .≈ correct_fields.v)
Stacktrace:
[1] run_ocean_large_eddy_simulation_regression_test(::GPU, ::VerstappenAnisotropicMinimumDissipation{Float64,Float64,Float64,Float64}) at /home/fpoulin/software/Oceananigans.jl/test/regression_tests/ocean_large_eddy_simulation_regression_test.jl:123
[2] macro expansion at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:77 [inlined]
[3] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
[4] macro expansion at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:74 [inlined]
[5] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
[6] top-level scope at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:60
Ocean large eddy simulation [GPU]: Test Failed at /home/fpoulin/software/Oceananigans.jl/test/regression_tests/ocean_large_eddy_simulation_regression_test.jl:124
Expression: all(test_fields.w .≈ correct_fields.w)
Stacktrace:
[1] run_ocean_large_eddy_simulation_regression_test(::GPU, ::VerstappenAnisotropicMinimumDissipation{Float64,Float64,Float64,Float64}) at /home/fpoulin/software/Oceananigans.jl/test/regression_tests/ocean_large_eddy_simulation_regression_test.jl:124
[2] macro expansion at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:77 [inlined]
[3] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
[4] macro expansion at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:74 [inlined]
[5] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
[6] top-level scope at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:60
[2020/11/13 11:12:48.923] INFO Testing oceanic large eddy simulation regression [SmagorinskyLilly, GPU]
[2020/11/13 11:13:08.185] INFO Δu: min=-5.669004e-10, max=+5.237555e-10, mean=-4.579272e-20, absmean=+3.847342e-12, std=+2.582632e-11 (4064/4096 matching grid points)
[2020/11/13 11:13:08.185] INFO Δv: min=-5.248542e-10, max=+4.306961e-10, mean=-4.446923e-20, absmean=+3.446188e-12, std=+2.026341e-11 (4081/4096 matching grid points)
[2020/11/13 11:13:08.185] INFO Δw: min=-8.810476e-10, max=+3.828646e-10, mean=-1.673779e-20, absmean=+2.695421e-12, std=+2.003712e-11 (3987/4096 matching grid points)
[2020/11/13 11:13:08.185] INFO ΔT: min=-3.171294e-10, max=+1.584823e-09, mean=+1.933831e-12, absmean=+3.283801e-12, std=+4.237465e-11 (4096/4096 matching grid points)
[2020/11/13 11:13:08.185] INFO ΔS: min=-5.826450e-13, max=+5.613288e-13, mean=-3.816392e-17, absmean=+3.587408e-15, std=+1.934380e-14 (4096/4096 matching grid points)
Ocean large eddy simulation [GPU]: Test Failed at /home/fpoulin/software/Oceananigans.jl/test/regression_tests/ocean_large_eddy_simulation_regression_test.jl:122
Expression: all(test_fields.u .≈ correct_fields.u)
Stacktrace:
Ocean large eddy simulation [GPU]: Test Failed at /home/fpoulin/software/Oceananigans.jl/test/regression_tests/ocean_large_eddy_simulation_regression_test.jl:122
Expression: all(test_fields.u .≈ correct_fields.u)
Stacktrace:
[1] run_ocean_large_eddy_simulation_regression_test(::GPU, ::VerstappenAnisotropicMinimumDissipation{Float64,Float64,Float64,Float64}) at /home/fpoulin/software/Oceananigans.jl/test/regression_tests/ocean_large_eddy_simulation_regression_test.jl:122
[2] macro expansion at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:77 [inlined]
[3] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
[4] macro expansion at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:74 [inlined]
[5] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
[6] top-level scope at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:60
Ocean large eddy simulation [GPU]: Test Failed at /home/fpoulin/software/Oceananigans.jl/test/regression_tests/ocean_large_eddy_simulation_regression_test.jl:123
Expression: all(test_fields.v .≈ correct_fields.v)
Stacktrace:
[1] run_ocean_large_eddy_simulation_regression_test(::GPU, ::VerstappenAnisotropicMinimumDissipation{Float64,Float64,Float64,Float64}) at /home/fpoulin/software/Oceananigans.jl/test/regression_tests/ocean_large_eddy_simulation_regression_test.jl:123
[2] macro expansion at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:77 [inlined]
[3] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
[4] macro expansion at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:74 [inlined]
[5] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
[6] top-level scope at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:60
Ocean large eddy simulation [GPU]: Test Failed at /home/fpoulin/software/Oceananigans.jl/test/regression_tests/ocean_large_eddy_simulation_regression_test.jl:124
Expression: all(test_fields.w .≈ correct_fields.w)
Stacktrace:
[1] run_ocean_large_eddy_simulation_regression_test(::GPU, ::VerstappenAnisotropicMinimumDissipation{Float64,Float64,Float64,Float64}) at /home/fpoulin/software/Oceananigans.jl/test/regression_tests/ocean_large_eddy_simulation_regression_test.jl:124
[2] macro expansion at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:77 [inlined]
[3] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
[4] macro expansion at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:74 [inlined]
[5] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
[6] top-level scope at /home/fpoulin/software/Oceananigans.jl/test/test_regression.jl:60
[2020/11/13 11:12:48.923] INFO Testing oceanic large eddy simulation regression [SmagorinskyLilly, GPU]
[2020/11/13 11:13:08.185] INFO Δu: min=-5.669004e-10, max=+5.237555e-10, mean=-4.579272e-20, absmean=+3.847342e-12, std=+2.582632e-11 (4064/4096 matching grid points)
[2020/11/13 11:13:08.185] INFO Δv: min=-5.248542e-10, max=+4.306961e-10, mean=-4.446923e-20, absmean=+3.446188e-12, std=+2.026341e-11 (4081/4096 matching grid points)
[2020/11/13 11:13:08.185] INFO Δw: min=-8.810476e-10, max=+3.828646e-10, mean=-1.673779e-20, absmean=+2.695421e-12, std=+2.003712e-11 (3987/4096 matching grid points)
[2020/11/13 11:13:08.185] INFO ΔT: min=-3.171294e-10, max=+1.584823e-09, mean=+1.933831e-12, absmean=+3.283801e-12, std=+4.237465e-11 (4096/4096 matching grid points)
[2020/11/13 11:13:08.185] INFO ΔS: min=-5.826450e-13, max=+5.613288e-13, mean=-3.816392e-17, absmean=+3.587408e-15, std=+1.934380e-14 (4096/4096 matching grid points)
Ocean large eddy simulation [GPU]: Test Failed at /home/fpoulin/software/Oceananigans.jl/test/regression_tests/ocean_large_eddy_simulation_regression_test.jl:122
Expression: all(test_fields.u .≈ correct_fields.u)
Stacktrace:
Ah nice doesn't look like there's much cause for concern as the regression test "mostly passes" (I copy pasted the relevant portion below).
Almost all the grid points match except for a few percent of the velocity grid points. So I'm guessing this is just a result of different GPUs doing calculations slightly differently maybe? Tiny differences could accumulate over the 100 iterations of the regression test.
I suspect this is the same reason why 2/166 checkpointer tests fail.
Maybe we need to add an absolute tolerance to these tests when checking for ≈
?
[2020/11/13 11:13:08.185] INFO Δu: min=-5.669004e-10, max=+5.237555e-10, mean=-4.579272e-20, absmean=+3.847342e-12, std=+2.582632e-11 (4064/4096 matching grid points)
[2020/11/13 11:13:08.185] INFO Δv: min=-5.248542e-10, max=+4.306961e-10, mean=-4.446923e-20, absmean=+3.446188e-12, std=+2.026341e-11 (4081/4096 matching grid points)
[2020/11/13 11:13:08.185] INFO Δw: min=-8.810476e-10, max=+3.828646e-10, mean=-1.673779e-20, absmean=+2.695421e-12, std=+2.003712e-11 (3987/4096 matching grid points)
[2020/11/13 11:13:08.185] INFO ΔT: min=-3.171294e-10, max=+1.584823e-09, mean=+1.933831e-12, absmean=+3.283801e-12, std=+4.237465e-11 (4096/4096 matching grid points)
[2020/11/13 11:13:08.185] INFO ΔS: min=-5.826450e-13, max=+5.613288e-13, mean=-3.816392e-17, absmean=+3.587408e-15, std=+1.934380e-14 (4096/4096 matching grid points)
@francispoulin Out of curiosity if you run the nvidia-smi
command at the terminal does it tell you which GPU(s) you have?
Sure thing. See below.
$ nvidia-smi
Fri Nov 13 13:40:39 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 435.21 Driver Version: 435.21 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro P400 Off | 00000000:65:00.0 On | N/A |
| 34% 39C P0 N/A / N/A | 755MiB / 1977MiB | 3% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1880 G /usr/lib/xorg/Xorg 428MiB |
| 0 2040 G /usr/bin/gnome-shell 187MiB |
| 0 2583 G ...AAAAAAAAAAAACAAAAAAAAAA= --shared-files 134MiB |
+-----------------------------------------------------------------------------+
I don't think I've ever tried running Julia on a workstation GPU like the Quadro P400.
We usually run with a Volta V100 or a Titan V which both share the same Volta microarchitecture so maybe this is why we haven't seen big differences between the two. But I think the P400 uses the Pascal microarchitecture. So this might support the claim that the tests are failing due to accumulation of tiny arithmetic differences between GPUs (since GPUs between microarchitectures might produce slightly different answers).
I plan to do much in the way of GPU computing on my desktop with the current system. If the fact that my GPU's are pretty much non-existant is the problem, then that's not a big deal. Just troubling when we see errors but I can ignore them.
I am okay with closing the ticket if this is due to my GPU lacking achitecture.
Ah I wouldn't say your GPU lacks any architecture, it's just different from the one we test on. And we don't have access to many different GPUs.
Might be good to leave this issue open until we increase the ≈ tolerance to account for different GPU microarchitectures (at which points tests should pass on your system).
Sounds good @ali-ramadhan and I agree that it would be nice to have the software work on various machines, not just the good ones. ;)
I think that using tolerances to measure this makes a lot of sense. I am happy to do testing whenever you like.
Our GPU CI runs on sverdrup at MIT, which has a Quadro P6000:
So we do run the regression tests there successfully. But P400 might be different than P6000.
As a side note we should run benchmarks on GPUs like the P400 and P6000, since I think they may actually benefit from Float32
(unlike calculations on Volta chips, which have plenty of double precision capability).
After the big changes yesterday I decided to run the tests to make sure everything was working. Thanks again @ali-ramadhan for helping me get that started.
Now I'm finding that there are some failures and somethings that are broken. See below. Is it just me or do others get this now?