tracel-ai / burn

Burn is a new comprehensive dynamic Deep Learning Framework built using Rust with extreme flexibility, compute efficiency and portability as its primary goals.
https://burn.dev
Apache License 2.0
8.5k stars 420 forks source link

Potential false positives in negative assertions of data equality #1982

Open antimora opened 2 months ago

antimora commented 2 months ago

Description

We have identified an issue with our tensor operation tests, particularly in how they handle data types. This inconsistency can lead to false positives in our test suite, potentially masking real issues in our tensor operations.

Current Behavior

  1. Some tests are using f64 data by default, while the test backend uses f32.
  2. The assert_eq() method checks for data type equality, which is good for positive assertions but can lead to false positives in negative assertions.
  3. When using assert_eq(&expected, false), we can't be sure what specifically failed (data values, dimensions, or data type).

Example

In crates/burn-tensor/src/tests/ops/slice.rs:

#[test]
fn clamp_when_slice_exceeds_dimension() {
    let data = TensorData::from([0.0, 1.0, 2.0]);  // This uses f64 by default
    let tensor = Tensor::<TestBackend, 1>::from_data(data.clone(), &Default::default());

    let output = tensor.slice([0..4]);

    output.into_data().assert_eq(&data, true);
}

This test fails with:

assertion `left == right` failed: Data types differ (F32 != F64)
  left: F32
 right: F64

Expected Behavior

  1. Consistent use of data types across tests and backends.
  2. Ability to perform negative assertions without risk of false positives due to type mismatches.

Proposed Solutions

  1. Standardize on a single data type (e.g., f32) across all tests, or explicitly specify the data type in each test.
  2. Consider adding separate methods for asserting value equality and type equality.

Additional Context

There are many tests that use assert_eq(&expected, false) which can give a false positive assertion, e.g. the data is equal but datatype is not but actually we want the data not to be equal.

laggui commented 2 months ago

@nathanielsimard remember when I first made the tests strict and had to convert every data type for correct comparison then we decided to make it less strict? Where do we want to go from here?

laggui commented 2 months ago

I think the first proposed solution is too restrictive, because not all backends support the same dtypes necessarily. And we might want to run tests in f16 at some point.

But adding separate methods for testing value equality and complete equality (w/ dtype) is a good idea.

Maybe something like .assert_values_eq(other) which does not test dtype and .assert_eq(other) which tests both values and dtype. We could also have inequality methods (so the same but _ne instead of _eq) for shorthand.