search

xtensor-stack / xtensor

C++ tensors with broadcasting and lazy computing
BSD 3-Clause "New" or "Revised" License
3.36k stars 398 forks source link

Systematic testing #2658

Open AntoinePrv opened 1 year ago

AntoinePrv commented 1 year ago

Tests typically use the xt::xarray, sometimes xt::xtensor, but it turns there are some subtle differences between all available containers/expressions/views that leads to error when switching one for another.

A few issues have been caught only when reported by users, who then have to wait for another release to get the fix. For instance

Since this may generate a lot of tests, we would need to have strategy to not run/compile all of them at the same time (perhaps that would be only in CI, in parallel).

I am unsure if testing with project in other repositories (language bindings) should be done here or there. Depending if the fix needs to be in xtensor or in the other repository, there will always be a case or PR deadlock.

JohanMabille commented 1 year ago

A way to cover most of the use cases from the bindings is to test:

This should catch 99% of the issues. Other issues would be in the downstream projects themselves.

tdegeus commented 1 year ago

I concur. I too have had my annoying small bugs or missing templates:

That is also why I proposed

What I was planning to do is to write a Python module with most of the xtensor's functions with types xtensor_fixed, xtensor, xarray, pytensor, and pyarray and then test every function against NumPy based on random input. That would cover at least part of what you list above. The only reason that I was hesitating is maintainability. How would we ensure that all possible functions are included?

JohanMabille commented 1 year ago

How would we ensure that all possible functions are included?

I'm not sure that we can; but that should not be a blocker. Also I'm not sure we need to tests with pytensor and pyarray (or that means the tests should leave in another repo), as long as we tests on "containers" with similar properties (like the signed int shapes).

AntoinePrv commented 1 year ago

@JohanMabille remember we also had that pytensor issue due to the fact that the shape was a view on the Python data.

tdegeus commented 1 year ago

An initial proposition for the Python binding : https://github.com/xtensor-stack/xtensor-python/pull/288 . It would be great if we can discuss this a bit.

Furthermore: