SuperGoodSoft / solidus_taxjar

Support for using TaxJar to handle tax calculations in Solidus
BSD 3-Clause "New" or "Revised" License
12 stars 12 forks source link

Refund feature spec fails sporadically on CI #242

Open forkata opened 1 year ago

forkata commented 1 year ago

What

A feature spec we recently introduced has started failing sporadically on CI. In order to get the build passing and unblock us from merging PRs we've marked this test as pending.

Why

Some initial investigation showed that on some runs on CI, the order we create through the factory is not getting tax adjustments for it's line items, which means when we create the return the reimbursement does not include that amount. The strange thing is that this is the only feature spec we have that fails this way.

Passing Run

screenshot_2023-07-19-22-21-44 796

Failing Run

screenshot_2023-07-19-22-50-30 460

benjaminwil commented 10 months ago

We've been unable to reproduce this locally but have SSHed into CI runners and run rspec --bisect to reveal that some seeds fail consistently--but only in CI.

But each consistently-failing bisect with seed gives us just two tests, the second test always being this refund spec. But the first test always being different (so far).

We think there is a test-order dependency somewhere we just don't have enough information to find out what exactly it could be.