Different behavior when running testing against Rinkeby

[feamcor]

Describe the Bug When running truffle v5 javascript tests against Rinkeby it doesn't catch the revert. When running the tests against Ganache, behavior is as expected.

Error Message

  1) Contract: Bileto
       should not open store for non-owner:
     AssertionError: Expected to fail with by owner, but failed with: StatusError: Transaction: 0x5b4dc57076030dc52c18e15410bccaa1962db7f636204b8222469e888651320d exited with an error (status 0).
     Please check that the transaction:
     - satisfies all conditions set by Solidity `require` statements.
     - does not trigger a Solidity `revert` statement.
      at fails (node_modules/truffle-assertions/index.js:138:13)
      at process.internalTickCallback (internal/process/next_tick.js:77:7)

Test Code

  it("should not open store for non-owner", async () => {
    await truffleAssert.reverts(
      __contract.openStore({
        from: __organizer1
      }),
      "by owner");
  });

Expected Behavior It should recognize that method reverted and consider the test successful.

Environment Information

• Truffle version: 5.0.1 (core: 5.0.1)
• Web3 version: x.x
• Solidity: 0.5.2 (solc-js)
• truffle-assertions version: 0.7.2
• ganache-cli version: 6.2.5 (ganache-core: 2.3.3)

truffle-config.js

    rinkeby: {
      provider: () => new HDWallet(
        metamaskSeedPhrase,
        `https://rinkeby.infura.io/v3/${infuraProjectId}`,
        rinkebyAddressIndex,
        rinkebyNumAddresses),
      network_id: 4,
      skipDryRun: true
    },

[rkalis]

Hi @feamcor

Thanks for opening this issue. You're the first person I've spoken with that runs their tests on Rinkeby as well, so good job on being thorough. You can never test enough, especially with smart contracts.

I haven't looked into testing with Rinkeby, so I'll have to see how I can make it work with Rinkeby. I definitely think this should be fully supported by truffle-assertions, so I will see if I can make it work some time soon.

Just one idea I have from the top of my head looking at the error message. It could be that Infura Rinkeby doesn't return the "revert reason", since this is a very new feature. Could you try removing the reason string from your assertion, and check whether that one does pass?

Like this:

  it("should not open store for non-owner", async () => {
    await truffleAssert.reverts(
      __contract.openStore({
        from: __organizer1
      }));
  });

It's not ideal, but if the revert reason string is not supported by the Ethereum node itself then this is the only way to make it work for now.

[feamcor]

Hi @rkalis, thank you so much for the workaroud... it worked! I had 18 test cases failing due to that. See below the current test log after applying the changes you suggested (please ignore the failing test case, it is on purpose). Cheers.

$ truffle test --network rinkeby
Using network 'rinkeby'.

  Contract: Bileto
store address: 0xA8cd043eAD1ac774cE48Aa9b0320b8FA7c026b8d
    ✓ should create store
    ✓ should not open store for non-owner (15897ms)
    ✓ should open store (11298ms)
    ✓ should not suspend store for non-owner (10244ms)
    ✓ should suspend store (10713ms)
    ✓ should re-open store (9781ms)
    ✓ should not create an event for non-owner (10339ms)
    ✓ should not create an event when organizer is a contract (10344ms)
    ✓ should not create an event without external ID (10298ms)
    ✓ should not create an event without name (10136ms)
    ✓ should not create an event with incentive greater than 100% (11196ms)
    ✓ should not create an event with no tickets available for sale (10303ms)
    ✓ should create an event (10968ms)
    ✓ should store event basic info accordingly (982ms)
    ✓ should init event sales info accordingly (1707ms)
    ✓ should not complete purchase sales not started yet (26841ms)
    ✓ should start ticket sales of an event (11803ms)
    ✓ should not complete purchase when quantity is zero (8481ms)
    ✓ should not complete purchase when there are not enough tickets (9915ms)
    ✓ should not complete purchase without external ID (10304ms)
    ✓ should not complete purchase without timestamp (10608ms)
    ✓ should not complete purchase without customer ID (10458ms)
    ✓ should not complete purchase value less than total (10352ms)
    ✓ should not complete purchase value more than total (10429ms)
    ✓ should complete a purchase (11465ms)
    ✓ should store purchase info accordingly (997ms)
    ✓ should suspend ticket sales of an event (17902ms)
    ✓ should end ticket sales of an event (10905ms)
    ✓ should cancel a purchase
    ✓ should refund a purchase
    ✓ should check-in customer
    ✓ should complete an event (10854ms)
    ✓ should settle an event (10898ms)
    1) should cancel an event

    Events emitted during test:
    ---------------------------

    EventSettled(_id: <indexed>, _extId: <indexed>, _by: <indexed>, _settlement: 90000000000000000)

    ---------------------------
    ✓ should not close store for non-owner (9435ms)
    ✓ should close store (11229ms)

  35 passing (9m)
  1 failing

  1) Contract: Bileto
       should cancel an event:
     Error: Transaction: 0xb1429a5af1b3b86326884215cac3a53126a944a408d404c32a6ed44076bde3b8 exited with an error (status 0).
     Please check that the transaction:
     - satisfies all conditions set by Solidity `require` statements.
     - does not trigger a Solidity `revert` statement.

      at StatusError.ExtendableError (/usr/local/lib/node_modules/truffle/build/webpack:/packages/truffle-error/index.js:10:1)
      at new StatusError (/usr/local/lib/node_modules/truffle/build/webpack:/packages/truffle-contract/lib/statuserror.js:31:1)
      at Object.receipt (/usr/local/lib/node_modules/truffle/build/webpack:/packages/truffle-contract/lib/handlers.js:117:1)
      at process.internalTickCallback (internal/process/next_tick.js:77:7)

[rkalis]

Good to hear that it worked for you @feamcor. I'll leave this issue open for a bit so I can look into testing on Rinkeby some more.

[rkalis]

I added a few test cases with your Rinkeby error messages to the tests for truffleAssert.reverts(). So I'll close the issue now.