FEST tests not running smoothly

[jim618]

Just ran the FEST tests in develop. 17 of the tests failed and I managed to crash the JVM. I'll add a screenshot with the results a little later (cannot upload them in Starbucks for some reason)

[jim618]

[gary-rowe]

Yes, there's still lots to do with the wallet mode refactoring.

[jim618]

After discussion offline I'll look at this.

[jim618]

FEST test update:

StandardFestTest - these all run successfully but I sometimes see JRE failures (as per raised issue). TrezorFestTest - 2 passed 10 failed. I saw a MBHD crash report whilst these were running. KeepKeyFestTest - there's still quite a lot to do on these I think - still some references to Trezor classes in them.

[jim618]

TrezorFestTest, running tests individually in the IDE: verifyCreateHardwareWallet_ColdStart : PASS verifyCreateHardwareWallet_WarmStart : PASS verifyUnlockHardwareWallet_WarmStart : PASS verifyUnlockHardwareWallet_ColdStart : PASS verifyReattachHardwareWallet : PASS verifyRestoreWithLocalBackup : PASS verifySendScreen : PASS

I'll stop there - as the tests are running ok individually it looks like something to do with either the test framework or the soft reset.

[jim618]

Remaining TrezorFestTests (each test run individually in IDE): verifySendRequestScreen : PASS verifyUnsupportedFirmware : FAIL verifyPaymentsScreen : PASS verifyUnsupportedConfiguration_Passphrase : FAIL

[jim618]

just did an update an reran the TrezorFestTests all together. I still get 2 passes, 10 failures.

[jim618]

Just reran the TrezorFestTests all together. I still get 2 passes, 10 failures.

There seems to be a mixup with the accessibility names of the labels and textareas. For instance, 'verifySendRequestScreen' is failing because there are two components labelled 'notes' on the PaymentRequestScreen and the FEST test is looking for one called 'notes.value'.

AccessibilityDecorator.apply just adds 'notes' so I am not sure where the '.value' should be added.

@gary-rowe Could you have a look at this please ?

[jim618]

Just reran the FEST tests: Standard: 1 failure: verifyCreateWallet_ro_RO_ColdStart Trezor: All tests pass. KeepKey, 5 tests pass, 4 fail (verifyRestoreWithLocalBackup, verifyCreateHardwareWallet_ColdStart, verifyUnlockHardwareWallet_ColdStart, verifyCreateHardwareWallet_WarmStart)

[gary-rowe]

OK, I think I'm about done with the FEST tests for this release. I've got all but one working now (create Trezor from cold start) which is caused by a dangling reference to MainController being maintained through a SOFT reset. This won't occur in the real world and the test works fine in isolation.

Ready for review and close.

[jim618]

I have rerun all the tests and and all run except for verifyCreateWallet_ro_RO_ColdStart I didn't get this to run in isolation.

I think this will do for this release - we have full coverage of the Trezor and KeepKey running fine.

Closing.