Fuzz testing

[andreaTP]

In Wazero they have integrated some interesting fuzz tests. We should eventually do something similar.

Reference: https://github.com/tetratelabs/wazero/blob/dc4f64adc97850a7b9e999efb3310ef83fe2766e/internal/integration_test/fuzz/README.md

[thomasdarimont]

jazzer Seems to be a useful and maintained library for fuzzing java libraries

[andreaTP]

That's correct, in this case, one idea was to explore re-using wasm-smith to rely on third-party implementation of it. But I haven't evaluated how much work would be to re-implement things in Jazzer.

[bhelx]

Agreed. I think we should look into this after we get the core specs passing. Will help us find the next level of bugs.

[andreaTP]

for reference, we can even check the return values of generated wasm programs and compare it with a reference implementation. In WaZero they are comparing compiler with interpreter results ...

[bhelx]

yeah, this is called a "test oracle" https://en.wikipedia.org/wiki/Test_oracle

I tried to do this with wasmtime and wasmer but i couldn't get their Java libraries to work 😆

[andreaTP]

I tried to do this

Applied in this repository? Do you know how we can integrate this into our current workflow? Or is it still a distraction from getting the basic specs passing at this point in time?

[bhelx]

Yes, I tried to do it very early on before I even wrote the ruby spec generator. I think we should make all the specs green before we look to implement fuzzing.