search

pytorch / functorch

functorch is JAX-like composable function transforms for PyTorch.
https://pytorch.org/functorch/
BSD 3-Clause "New" or "Revised" License
1.38k stars 102 forks source link

[testing] Insufficient coverage in test suite #1090

Open kshitij12345 opened 1 year ago

kshitij12345 commented 1 year ago

In functorch test suite, we use sample_inputs to get samples from an OpInfo. The problem is that sample_inputs may or may not cover all the case/overloads for an operator. I think we should use reference_inputs which super set of sample_inputs and more comprehensive. (Though this will increase the test times).

Switching sample_inputs to reference_inputs leads to bunch of failure for test_op_has_batch_rule including the ones mentioned in https://github.com/pytorch/functorch/issues/1080 https://github.com/pytorch/functorch/issues/1069

Refer to https://github.com/pytorch/pytorch/pull/91355 for failures.

cc: @zou3519

zou3519 commented 1 year ago

Good catch!

Do we have a sense of how much this increases testing time by?

kshitij12345 commented 1 year ago

Will confirm this in a day or two.

kshitij12345 commented 1 year ago

Timings are posted here : https://github.com/pytorch/pytorch/pull/91355#issue-1509073244

lezcano commented 1 year ago

This one should probably be closed

kshitij12345 commented 1 year ago

I want to check other tests in functorch/test_ops.py like test_vmapvjp and friends. So keeping it open till then.