Too aggressive LN SM block height polling

[elsirion]

Idk if there were multiple state machines running or if we sometimes have a tight loop and then wait 1s again, either way: when waiting for a block height we can probably just poll ever 5-10 minutes or so.

2024-05-01T17:39:09.288276Z     ERROR   fedimint_ln_client::pay Error waiting for block height: 842165 FederationError { general: Some(Received errors from 2 peers: peer-0: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname, peer-1: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname), peers: {PeerId(0): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname)), PeerId(1): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname))} }     
2024-05-01T17:39:10.289802Z     ERROR   fedimint_ln_client::pay Error waiting for outgoing contract to be cancelled: FederationError { general: Some(Received errors from 2 peers: peer-0: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname, peer-1: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname), peers: {PeerId(0): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname)), PeerId(1): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname))} }       
2024-05-01T17:39:10.292801Z     ERROR   fedimint_ln_client::pay Error waiting for block height: 842164 FederationError { general: Some(Received errors from 2 peers: peer-0: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname, peer-1: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname), peers: {PeerId(0): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname)), PeerId(1): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname))} }     
2024-05-01T17:39:10.296077Z     ERROR   fedimint_ln_client::pay Error waiting for outgoing contract to be cancelled: FederationError { general: Some(Received errors from 2 peers: peer-0: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname, peer-1: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname), peers: {PeerId(0): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname)), PeerId(1): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname))} }       
2024-05-01T17:39:10.298122Z     ERROR   fedimint_ln_client::pay Error waiting for block height: 842165 FederationError { general: Some(Received errors from 2 peers: peer-0: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname, peer-1: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname), peers: {PeerId(0): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname)), PeerId(1): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname))} }     
2024-05-01T17:39:10.299508Z     ERROR   fedimint_ln_client::pay Error waiting for outgoing contract to be cancelled: FederationError { general: Some(Received errors from 2 peers: peer-0: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname, peer-1: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname), peers: {PeerId(0): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname)), PeerId(1): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname))} }       
2024-05-01T17:39:10.301532Z     ERROR   fedimint_ln_client::pay Error waiting for block height: 842165 FederationError { general: Some(Received errors from 2 peers: peer-0: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname, peer-1: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname), peers: {PeerId(0): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname)), PeerId(1): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname))} }     
2024-05-01T17:39:11.299572Z     ERROR   fedimint_ln_client::pay Error waiting for outgoing contract to be cancelled: FederationError { general: Some(Received errors from 2 peers: peer-0: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname, peer-1: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname), peers: {PeerId(0): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname)), PeerId(1): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname))} }       
2024-05-01T17:39:11.302180Z     ERROR   fedimint_ln_client::pay Error waiting for block height: 842164 FederationError { general: Some(Received errors from 2 peers: peer-0: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname, peer-1: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname), peers: {PeerId(0): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname)), PeerId(1): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname))} }     
2024-05-01T17:39:11.304346Z     ERROR   fedimint_ln_client::pay Error waiting for outgoing contract to be cancelled: FederationError { general: Some(Received errors from 2 peers: peer-0: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname, peer-1: Rpc error: Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname), peers: {PeerId(0): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname)), PeerId(1): Rpc(Transport(Failed to resolve IP addresses for this hostname: failed to lookup address information: No address associated with hostname))} }       

[bradleystachurski]

Bumping to 5-10 minutes sounds reasonable. Alternatively, would it make sense to switch to websockets? This may help fedimint-ln-client in other areas we poll (recently highlighted in this discord thread).

[elsirion]

Looking at it again, we already seem to use the wait_block_height fn (in await_contract_timeout), so what we actually need is some rate-limiting for re-trying on (connection) error. I think #4929 by @dpc already solved this, so a fix will be in 0.4.

https://github.com/fedimint/fedimint/blob/ec1bcf41b199f9314249b03bc498ee07a99b481b/modules/fedimint-ln-client/src/pay.rs#L468-L482

[justinmoon]

dev call: @dpc should validate that https://github.com/fedimint/fedimint/pull/4929 fixed this.

[dpc]

4929 should help with excessive retries on errors.