byte-physics / igortest

Igor Pro Universal Testing Framework
https://docs.byte-physics.de/igor-unit-testing-framework/
BSD 3-Clause "New" or "Revised" License
7 stars 2 forks source link

Allow rerun of flaky test cases #413

Closed Garados007 closed 1 year ago

Garados007 commented 1 year ago

Some test cases can fail from time to time and only need to be rerun some times to succeed.

Close #164

Garados007 commented 1 year ago

@t-b @MichaelHuth Ready for review.

t-b commented 1 year ago

Review:

Nice addition, works mostly!

I've tried with

Function/WAVE GetWaveA()

    Make/FREE wv = {100, 200}

    return wv
End

Function/WAVE GetWaveB()

    Make/FREE/T wv = {"a", "b"}

    return wv
End

// IUTF_RETRY_FAILED
// IUTF_TD_GENERATOR v0:GetWaveA
// IUTF_TD_GENERATOR s0:GetWaveB
static Function UnreliableStatus([STRUCT IUTF_mData &m])
    NVAR counter = root:retryCounter
    counter += 1
    CHECK_GE_VAR(counter, 5)

    print m.v0, m.s0
End

which works.

Garados007 commented 1 year ago

Applied the requested changes.

Can we make it work with REQUIRE as well?

Yes. I have added the support for this. It was a bit tricky as failed REQUIREs works differently from a failed CHECK as it aborts the execution right away. I made the support for rerunning failed REQUIREs optionally as there are other conditions that could abort the test case. I updated the documentation and recommend the use of CHECKs for flaky conditions.