tracel-ai / burn

Burn is a new comprehensive dynamic Deep Learning Framework built using Rust with extreme flexibility, compute efficiency and portability as its primary goals.
https://burn.dev
Apache License 2.0
9.07k stars 450 forks source link

Precision errors running tests #1477

Open McArthur-Alford opened 8 months ago

McArthur-Alford commented 8 months ago

Describe the bug When running the tests, I run into some precision errors. I have ran them a handful of times, and they seem to consistently be the following two tests:

tests::jit::module_nearest_interpolate::tests::test_downsample_interpolation
tests::jit_fusion::module_nearest_interpolate::tests::test_downsample_interpolation

The full log is:

failures:

---- tests::jit::module_nearest_interpolate::tests::test_downsample_interpolation stdout ----
thread 'tests::jit::module_nearest_interpolate::tests::test_downsample_interpolation' panicked at crates/burn-wgpu/src/lib.rs:73:5:
Tensors are not approx eq:
  => Position 3: 7 != 6 | difference 1 > tolerance 0.0010000000000000002
  => Position 9: 161 != 160 | difference 1 > tolerance 0.0010000000000000002
  => Position 15: 315 != 314 | difference 1 > tolerance 0.0010000000000000002
  => Position 21: 469 != 468 | difference 1 > tolerance 0.0010000000000000002

---- tests::jit_fusion::module_nearest_interpolate::tests::test_downsample_interpolation stdout ----
thread 'tests::jit_fusion::module_nearest_interpolate::tests::test_downsample_interpolation' panicked at crates/burn-wgpu/src/lib.rs:73:5:
Tensors are not approx eq:
  => Position 3: 7 != 6 | difference 1 > tolerance 0.0010000000000000002
  => Position 9: 161 != 160 | difference 1 > tolerance 0.0010000000000000002
  => Position 15: 315 != 314 | difference 1 > tolerance 0.0010000000000000002
  => Position 21: 469 != 468 | difference 1 > tolerance 0.0010000000000000002

To Reproduce

  1. Clone the latest version of main
  2. Use run-checks.sh and wait for the errors

Hardware:

WorldSEnder commented 8 months ago

As I noted in #1444 I ignored these tests too on my local run. Hardware:

Arch Linux, kernel on 6.7.9 Intel i7-13700K AMD Radeon RX 5700 XT

antimora commented 8 months ago

@nathanielsimard since you guys are working on this