[BUG] failing tests do not fail our CI/CD gh action

[austinkline]

Current Behavior

Our gh action which runs our test suite doesn't detect a failure properly. If you make a PR with failing tests, the action itself will still look like it passed

Expected Behavior

A failing test should abort the test and propagate that failure up to the action

Steps To Reproduce

1. Write a test that always fails
2. Make a PR
3. GH action will pass

Environment

N/A

What are you currently working on that this is blocking?

N/A

[austinkline]

Looks like this is because the flow test runner isn't returning a 1 exit code, and instead returns a 0

hybrid-custody git:(ci-cd) flow test --cover ./test/*_tests.cdc

❗   Version warning: a new version of Flow CLI is available (v1.2.0).
   Read the installation guide for upgrade instructions: https://docs.onflow.org/flow-cli/install

Test results: "./test/CapabilityFactory_tests.cdc"
- PASS: testGetProviderCapability
Test results: "./test/CapabilityProxy_tests.cdc"
- PASS: testSetupProxy
- FAIL: testShareExampleNFTCollectionPublic
        Execution failed:
            error: assertion failed: failed to borrow proxy
               --> 7465737400000000000000000000000000000000000000000000000000000000:244:4

- FAIL: testShareExampleNFTCollectionPrivate
        Execution failed:
            error: internal error: open scripts/proxy/get_all_private_capts.cdc: no such file or directory
            goroutine 1 [running]:
            ...
<TRUNCATED>

               --> 7465737400000000000000000000000000000000000000000000000000000000:193:4

Test results: "./test/HybridCustody_tests.cdc"
- PASS: testImports
- PASS: testSetupFactory
- PASS: testSetupFactoryWithFT
- PASS: testSetupChildAccount
- PASS: testPublishAccount
- PASS: testRedeemAccount
- PASS: testProxyAccount_getAddress
- PASS: testProxyAccount_hasChildAccounts
- PASS: testProxyAccount_getFTCapability
- PASS: testProxyAccount_getCapability
- PASS: testProxyAccount_getPublicCapability
- PASS: testCheckParentRedeemedStatus
- PASS: testSeal
- PASS: testTransferOwnership
- PASS: testGetCapability_ManagerFilterAllowed
- PASS: testGetCapability_ManagerFilterNotAllowed
- PASS: testGetPrivateCapabilityFromProxy
- PASS: testGetPublicCapabilityFromProxy
- PASS: testMetadata_ProxyAccount_Metadata
- PASS: testMetadata_ChildAccount_Metadata
- PASS: testGetAddresses
- PASS: testRemoveChildAccount
- PASS: testGetAllFlowBalances
- PASS: testGetAllFTBalance
- PASS: testGetFlowBalanceByStoragePath
- PASS: testGetNFTDisplayViewFromPublic
- PASS: testGetAllCollectionViewsFromStorage
- PASS: testSetupChildAndParentMultiSig
- PASS: testSendChildFtsWithParentSigner
- PASS: testAddExampleTokenToBalance
- PASS: testSetupChildWithDisplay
- PASS: testGetProxyAccountNFTCapabilities
- PASS: testGetNFTsAccessibleFromProxyAccount
- PASS: testGetProxyAccountFTCapabilities
- PASS: testBlockchainNativeOnboarding
Coverage: 72.8% of statements

➜  hybrid-custody git:(ci-cd) echo $?
0

[austinkline]

Even when splitting them up, the issue is not solved unfortunately:

#!/bin/bash

set -e

flow test --cover ./test/CapabilityFactory_tests.cdc
flow test --cover ./test/CapabilityProxy_tests.cdc
flow test --cover ./test/HybridCustody_tests.cdc

The above script has the same outcome:

➜  hybrid-custody git:(ci-cd) ✗ echo $?
0

[austinkline]

The flow cli has a new published version which is supposed to have fixed this, verifying and then I'll get a PR out

[austinkline]

➜  hybrid-custody git:(ci-cd) ✗ echo $?
1