Reduce time spent in CI

Summary

Test	Time taken	Action	Potential savings
`test_atomic_ibc_forward_hop_transfer`	> 720s	Reduce time	600s
`test_misspelled_memo_content_ibc_forward_transfer`	> 420s	Reduce time	300s
`test_misspelled_memo_fields_ibc_forward_transfer`	> 420s	Reduce time	300s
`test_filter_incentivized_fees_relayer`	> 300s	Reduce time	180s
`test_pay_packet_fee_async`	> 300s	Reduce time	180s
`test_ternary_ibc_transfer`	> 420s	Reduce time	300s
`python_end_to_end_tests`	> 240s	Remove test	240s
Total:	> 2820s		2100s

Total time: 47min

If we can get all of those under 120s we'd save ~35min and get to ~12min.

Tasks

List of tasks to reduce the time spent in CI:

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate milestone (priority) applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

Here are some thoughts on the mentioned tests, let me know what you think.

Tests with more than 2 chains

The time for these tests is mostly due to the Connection and Channel handshakes:

test_atomic_ibc_forward_hop_transfer need 4 chains to run, as it will test forwarding a packet from A to D, passing by B and C. The bootstrapping will fully connect chains A, B, C and D
test_misspelled_memo_content_ibc_forward_transfer and test_misspelled_memo_fields_ibc_forward_transfer bootstrap chains A, B and C
test_ternary_ibc_transfer bootstraps chains A, B and C

For this reducing the handshake times should decrease the run time of all tests.

ICS29 tests

test_filter_incentivized_fees_relayer
test_pay_packet_fee_async

Some ICS29 tests take longer due to a change introduced in v8.1.0+, where only the highest value between recv+ack and timeout fee is escrowed. First verifies if the max(recv+ack, timeout) is escrowed and if it fails after 90 seconds, verifies if recv+ack+timeout is escrowed. This allows the test to be compatible with both implementation.

For this solution we could either verify if the amount a or b is escrowed at each iteration, or add a feature flag to determine if the chain is running v8.1.0+ or not.

CLI tests

The python_end_to_end_tests is testing many CLIs. This takes a long time as it contains many tests grouped in a single one. Not sure if there is any way to easily reduce the time for this one, but it might be a good idea to migrate the tests to use the test framework so that we can remove the python dependency.

informalsystems / hermes