Using ”as_cell_slice“ for same matrix results in inconsistent matrixes

[dbsxdbsx]

with system Ubuntu 20.04.3 LTS:

[dependencies]
tch = "0.5.0"

[dev-dependencies]
numpy = "0.14.1"
ndarray = "0.15.3"
pyo3 = { version = "0.14.1", features = ["macros", "auto-initialize"] }
serial_test = "0.5.1"

I am doing some test by checking whether the numpy object from python is the same as the one produced by rust code. Core code like this:

    // let tmp_rust_side_array = &t.to_kind(tch::Kind::Float);
    let rust_side_array: ArrayD<f64> = t.try_into().unwrap();
    let rust_side_array: &PyArrayDyn<f64> = rust_side_array.to_pyarray(py); ////or default type:&PyArrayDyn<f64,Dim<IxDynImpl>>
    println!("rust pyarray1:{}", rust_side_array);
    println!("python pyarray1:{}", python_side_array); // type:&PyArrayDyn<f64>
    println!("===========1===========\n");
    let python_side_array = python_side_array.as_cell_slice().unwrap();
    let rust_side_array = rust_side_array.as_cell_slice().unwrap();
    println!("rust pyarray2:{:?}", rust_side_array);
    println!("python pyarray2:{:?}", python_side_array);
    println!("===========2===========\n");
    assert_eq!(rust_side_array, python_side_array,);
    println!("===========after assertion===========\n");

For this code, every assersion is passed until it comes with this matrix (printed by rust pyarray1):

rust pyarray1:[[0. 0. 2.]
 [0. 1. 2.]
 [1. 1. 0.]]
python pyarray1:[[0. 0. 2.]
 [0. 1. 2.]
 [1. 1. 0.]]
===========1===========

rust pyarray2:[Cell { value: 0.0 }, Cell { value: 0.0 }, Cell { value: 2.0 }, Cell { value: 0.0 }, Cell { value: 1.0 }, Cell { value: 2.0 }, Cell { value: 1.0 }, Cell { value: 1.0 }, Cell { value: 0.0 }]
python pyarray2:[Cell { value: 0.0 }, Cell { value: 0.0 }, Cell { value: 1.0 }, Cell { value: 0.0 }, Cell { value: 1.0 }, Cell { value: 1.0 }, Cell { value: 2.0 }, Cell { value: 2.0 }, Cell { value: 0.0 }]
===========2===========

thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `[Cell { value: 0.0 }, Cell { value: 0.0 }, Cell { value: 2.0 }, Cell { value: 0.0 }, Cell { value: 1.0 }, Cell { value: 2.0 }, Cell { value: 1.0 }, Cell { value: 1.0 }, Cell { value: 0.0 }]`,
 right: `[Cell { value: 0.0 }, Cell { value: 0.0 }, Cell { value: 1.0 }, Cell { value: 0.0 }, Cell { value: 1.0 }, Cell { value: 1.0 }, Cell { value: 2.0 }, Cell { value: 2.0 }, Cell { value: 0.0 }]`', tests/against_python.rs:106:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
FAILED

The matrix from both rust and python are the same, but after as_cell_slice, they are of DIFFERENT order.

I can not figure out what is wrong there, I've check if this is related to orignal type, seems not.

By the way, I want to know how to compare 2 arrays if there is NaN. I want to treat NaN as equivalence.

[davidhewitt]

Without being able to see your code, I would guess this could be due to fortran-style vs c-style row ordering?

If the matrix produced is a transpose, then it's possible the slice will look different (slice can't take into account strides other than 1).

[dbsxdbsx]

@davidhewitt , the code is here:https://github.com/spebern/tch-distr-rs/blob/master/tests/against_python.rs. when change code on line 885 to test_cases.sample = Some(vec![vec![1], vec![3]]); and test the categorical block, the issue occurs.

We tested it on ubuntu 20.

[davidhewitt]

I'm afraid I don't have time to try to evaluate all that code; can you add this to your evaluation:

println!("rust pyarray state: {:?} {:?} {:?}", rust_side_array.is_c_contiguous(), rust_side_array.is_fortran_contiguous(), rust_side_array.strides());
println!("python pyarray state: {:?} {:?} {:?}", python_side_array.is_c_contiguous(), python_side_array.is_fortran_contiguous(), python_side_array.strides());

If my guess is correct, you will see that these arrays are laid out differently.

[dbsxdbsx]

@davidhewitt ,thanks for your advice, I think you are right. Now, before comparison, I force array to be contiguous, and now no error occurs. code like this:

python_side_array.call_method0("contiguous")
            .unwrap()
            .call_method0("numpy")
            .unwrap()
            .extract()
            .unwrap()

And several days ago I also found that, for some case, if the array is not contiguous, they can not be called with as_cell_slice.