search

webonnx / wonnx

A WebGPU-accelerated ONNX inference run-time written 100% in Rust, ready for native and the web
Other
1.61k stars 59 forks source link

Add testing framework to compare against known-good implementations #30

Open pixelspark opened 2 years ago

pixelspark commented 2 years ago

Is your feature request related to a problem? Please describe.

In order to be able to test correctness of the implementation it would be a good idea to be able to automatically compare it to some other known-good reference point (ideally the ONNX test suites but less ideally some other mature implementation, i.e. https://github.com/sonos/tract for instance).

Describe the solution you'd like

This could simply be a test that runs a set of ONNX models with specific inputs and outputs in WONNX and some other runtime, and then compares the result. I have tested this approach already here: https://github.com/pixelspark/nnx/blob/main/src/main.rs#L126 (and here's how to do inference with tract).

Describe alternatives you've considered

Well, writing tests that check every corner case by reading the spec very carefully :-)

haixuanTao commented 2 years ago

That's a great idea! It will reduce error and devs!

pixelspark commented 2 years ago

@haixuanTao One other interesting thing is that I believe tract can also do the inference of dimensions (we require onnx-simplifier), might want to have a look at how they are doing it!

pixelspark commented 2 years ago

There is also an 'official' ONNX test harness, which apparently you can plug your own backend in to test it: https://github.com/onnx/onnx/blob/master/docs/ImplementingAnOnnxBackend.md. Probably requires some Python glue code.

haixuanTao commented 2 years ago

Ok, so, I think I'm going to plug to the backend tester with python and put it in the CI :)

haixuanTao commented 2 years ago

Just leaving a message here, that I'm working on implementing the onnx backend test suite. But I'm in the middle of a professional move at the moment that is taking a bit of time 😅

I'm going to have more time in February 🙂