SP2 failing all scenarios

[konradkonrad]

On May 19th, all scenario player runs at SP2 failed with the same/similiar message:

Traceback (most recent call last):
  File "/scenario-player/scenario_player/tasks/base.py", line 140, in __call__
    return_val = self._run(*args, **kwargs)
  File "/scenario-player/scenario_player/tasks/execution.py", line 36, in _run
    task()
  File "/scenario-player/scenario_player/tasks/base.py", line 140, in __call__
    return_val = self._run(*args, **kwargs)
  File "/scenario-player/scenario_player/tasks/execution.py", line 36, in _run
    task()
  File "/scenario-player/scenario_player/tasks/base.py", line 140, in __call__
    return_val = self._run(*args, **kwargs)
  File "/scenario-player/scenario_player/tasks/api_base.py", line 59, in _run
    f'HTTP status code "{resp.status_code}" while fetching {url}. '
scenario_player.exceptions.RESTAPIStatusMismatchError: HTTP status code "402" while fetching http://127.0.0.1:39457/api/v1/channels. Expected 201: {"errors": "The account balance is below the estimated amount necessary to finish the lifecycles of all active channels. A balance of at least 1033700486102393600 wei is required."}

[konradkonrad]

The SP wallet on SP2 has currently 80+ goerliETH

[konradkonrad]

I'm comparing 20210519T182817_bf1_basic_functionality@SP1 and 20210519T182947_bf2_long_running@SP2 and both of them use comparable gas prices:

gas_price=6304000000 vs gas_price=6305000000

SP1
root@kronk:~# docker logs 20210519T182947_bf2_long_running|& grep gas_price|tail -1
2021-05-19 18:30:36.211993 [debug    ] Transaction sent               [raiden.network.rpc.client] args=(b'\x14\xe0DT}Mw\xe1\x0e\xac4\xfa\x890\x14^D\xb8\xbe\x05', 5500000000000000000) chain_id=5 eth_node=<EthClient.PARITY: 'parity'> from_address=0x8f5EA3Bb3E99C317b8745918F6973f7AfBF659c3 function_name=deposit gas_price=6304000000 given_block_identifier=0x563e81f732b3d9f0fdfb8fda71af669bcf9f0e84ecbe25e5cf7d3048e6196cf8 kwargs={} node=0x8f5EA3Bb3E99C317b8745918F6973f7AfBF659c3 nonce=169807 previous_total_deposit=0 startgas=127994 to_address=0xD145252daB9323F9Acb70295d6a0589c611F225d token=4ce2bc3a-61db-4c76-a4e0-ddd6778b3f8c transaction_hash=0x17207b8a27d1d3a54bd61fb038f34c6a054758cac2c82595facc18422d35a4f5 value=0

---

SP2
root@yzma:~# docker logs 20210519T182817_bf1_basic_functionality|& grep gas_price|tail -1
2021-05-19 18:29:03.393519 [debug    ] Transaction sent               [raiden.network.rpc.client] args=(10000000000000000000, '0x305c8d944158B97140334b0746e04EB8b7e94140') chain_id=5 eth_node=<EthClient.PARITY: 'parity'> from_address=0x9202584Ac2A5081C6d1F27d637d1DD1Fb2AEc6B7 function_name=mintFor gas_price=6305000000 kwargs={} node=0x9202584Ac2A5081C6d1F27d637d1DD1Fb2AEc6B7 nonce=88662 startgas=66478 to_address=0x59105441977ecD9d805A4f5b060E34676F50F806 token=de3c3476-a1d0-4d6b-bcbf-b3b4ec03d18b transaction_hash=0x46dfc6779208ddfb5503d70331be54545f9157db8500d3e9e7e63456feda2581 value=0

[konradkonrad]

At the time of the error, the raiden client expected its account to have a balance of

In [5]: Web3.fromWei(1033700486102393600, "ether")                                                                             
Out[5]: Decimal('1.0337004861023936')

around 1 ETH. That is of course way above the 0.3 ETH transferred at the beginning of the scenario.

The question remains: why would the raiden nodes on SP1 not throw the same error at the same time?

[konradkonrad]

That is the list of failed scenario containers:

docker ps -a --no-trunc|grep -v "lc/raiden"|grep 20210519|cut -b571-
20210519T191945_pfs9_partial_withdraw
20210519T191733_pfs6_simple_path_rewards
20210519T191501_pfs3_multiple_paths
20210519T191232_pfs1_get_a_simple_path
20210519T191034_ms3_simple_monitoring
20210519T190835_ms2_simple_monitoring
20210519T190602_mfee4_combined_fees
20210519T190339_mfee1_flat_fee
20210519T185821_bf7_long_path
20210519T185418_bf6_stress_hub_node
20210519T185131_bf4_multi_payments_same_node
20210519T184907_bf3_multi_directional_payment
20210519T182947_bf2_long_running

[karlb]

This happens on both SP1 and SP2 now.

[karlb]

We are giving 0.3 ETH to each node, but the errors indicate that more ETH (1.03 in the case of the error in the top message) is needed.

[karlb]

Goerli had frequent congestion issues recently. With that in mind, theses errors might be reasonable, depending on the actual gas prices at the time. I can't find a data source for goerli gas prices, so this is still mostly speculation.

[istankovic]

@karlb, @konradkonrad is this issue still relevant?

[karlb]

This really was just the goerli network having problems.