search

cosmology-tech / starship

A k8s based unified development environment for Cosmos Ecosystem (and beyond)
https://cosmology.zone/products/starship
MIT License
66 stars 20 forks source link

bug: Can't relay packets between chains #490

Open Kayanski opened 1 week ago

Kayanski commented 1 week ago

When using Starship with the following configuration (Config below) I can create a channel but the packets inside the channel are not relayed. I also paste the hermes-juno-stargaze logs from the channel creation (Hermes Pod Logs below).

I'm using a channel for Polytone which is a wasm to wasm channel

When I run hermes start the packet gets relayed correctly without issues

Expected behavior

I want starship to be able to create a channel and then relay the packets on it automatically without having to run hermes start manually

Investigation

Config: 

chains:
  - id: stargaze-1
    name: stargaze
    numValidators: 1
    ports:
      rest: 1313
      rpc: 26653
      grpc: 30658
      faucet: 8000
  - id: juno-1
    name: juno
    numValidators: 1
    ports:
      rest: 1317
      rpc: 26657
      grpc: 30657
      faucet: 8001

relayers:
  - name: osmo-juno
    type: hermes
    replicas: 1
    chains:
      - stargaze-1
      - juno-1

explorer:
  enabled: true
  ports:
    rest: 8080

registry:
  enabled: true
  ports:
    rest: 8081

Hermes Pod Logs: 

2024-06-17T09:28:35.621877Z  INFO ThreadId(392) worker.batch{chain=juno-1}:supervisor.handle_batch{chain=juno-1}:supervisor.process_batch{chain=juno-1}:worker.channel{channel=channel::channel-1/wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8:juno-1->stargaze-1}: 🎊  stargaze-1 => OpenTryChannel(OpenTry { port_id: wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3, channel_id: channel-1, connection_id: connection-0, counterparty_port_id: wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8, counterparty_channel_id: channel-1 }) at height 1-376
2024-06-17T09:28:35.621907Z  INFO ThreadId(392) worker.batch{chain=juno-1}:supervisor.handle_batch{chain=juno-1}:supervisor.process_batch{chain=juno-1}:worker.channel{channel=channel::channel-1/wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8:juno-1->stargaze-1}: channel handshake step completed with events: OpenTryChannel(OpenTry { port_id: wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3, channel_id: channel-1, connection_id: connection-0, counterparty_port_id: wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8, counterparty_channel_id: channel-1 })
2024-06-17T09:28:36.350994Z ERROR ThreadId(08) worker.batch{chain=stargaze-1}:supervisor.handle_batch{chain=stargaze-1}:supervisor.process_batch{chain=stargaze-1}:worker.channel{channel=channel::channel-1/wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3:stargaze-1->juno-1}:send_messages_and_wait_commit{chain=juno-1 tracking_id=ChannelOpenAck}:send_tx_with_account_sequence_retry{chain=juno-1 account.sequence=10}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call `send_tx_simulate` failed with status: status: Unknown, message: "account sequence mismatch, expected 11, got 10: incorrect account sequence [cosmos/cosmos-sdk@v0.47.5/x/auth/ante/sigverify.go:269] With gas wanted: '100000000000' and gas used: '52577' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "155"} }
2024-06-17T09:28:36.351036Z  WARN ThreadId(08) worker.batch{chain=stargaze-1}:supervisor.handle_batch{chain=stargaze-1}:supervisor.process_batch{chain=stargaze-1}:worker.channel{channel=channel::channel-1/wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3:stargaze-1->juno-1}:send_messages_and_wait_commit{chain=juno-1 tracking_id=ChannelOpenAck}:send_tx_with_account_sequence_retry{chain=juno-1 account.sequence=10}: failed to estimate gas because of a mismatched account sequence number, refreshing account sequence number and retrying once error=gRPC call `send_tx_simulate` failed with status: status: Unknown, message: "account sequence mismatch, expected 11, got 10: incorrect account sequence [cosmos/cosmos-sdk@v0.47.5/x/auth/ante/sigverify.go:269] With gas wanted: '100000000000' and gas used: '52577' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "155"} }
2024-06-17T09:28:36.352655Z  INFO ThreadId(08) worker.batch{chain=stargaze-1}:supervisor.handle_batch{chain=stargaze-1}:supervisor.process_batch{chain=stargaze-1}:worker.channel{channel=channel::channel-1/wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3:stargaze-1->juno-1}:send_messages_and_wait_commit{chain=juno-1 tracking_id=ChannelOpenAck}:send_tx_with_account_sequence_retry{chain=juno-1 account.sequence=10}: refreshed account sequence number old=10 new=11
2024-06-17T09:28:36.958678Z  INFO ThreadId(411) worker.batch{chain=stargaze-1}:supervisor.handle_batch{chain=stargaze-1}:supervisor.process_batch{chain=stargaze-1}:worker.channel{channel=channel::channel-1/wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3:stargaze-1->juno-1}: 🎊  juno-1 => OpenAckChannel(OpenAck { port_id: wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8, channel_id: channel-1, connection_id: connection-0, counterparty_port_id: wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3, counterparty_channel_id: channel-1 }) at height 1-156
2024-06-17T09:28:36.958711Z  INFO ThreadId(411) worker.batch{chain=stargaze-1}:supervisor.handle_batch{chain=stargaze-1}:supervisor.process_batch{chain=stargaze-1}:worker.channel{channel=channel::channel-1/wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3:stargaze-1->juno-1}: channel handshake step completed with events: OpenAckChannel(OpenAck { port_id: wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8, channel_id: channel-1, connection_id: connection-0, counterparty_port_id: wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3, counterparty_channel_id: channel-1 })
2024-06-17T09:28:36.970208Z  INFO ThreadId(08) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}:foreign_client.detect_misbehaviour{client=juno-1->stargaze-1:07-tendermint-0}: No evidence of misbehavior detected for chain juno-1
2024-06-17T09:28:37.070329Z  INFO ThreadId(10) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}: client is valid
2024-06-17T09:28:37.828027Z  INFO ThreadId(07) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=stargaze-1->juno-1:07-tendermint-0}:foreign_client.detect_misbehaviour{client=stargaze-1->juno-1:07-tendermint-0}: No evidence of misbehavior detected for chain stargaze-1
2024-06-17T09:28:37.880447Z  INFO ThreadId(08) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}:foreign_client.detect_misbehaviour{client=juno-1->stargaze-1:07-tendermint-0}: No evidence of misbehavior detected for chain juno-1
2024-06-17T09:28:37.928145Z  INFO ThreadId(13) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=stargaze-1->juno-1:07-tendermint-0}: client is valid
2024-06-17T09:28:37.980560Z  INFO ThreadId(10) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}: client is valid
2024-06-17T09:28:38.989478Z ERROR ThreadId(07) worker.batch{chain=juno-1}:supervisor.handle_batch{chain=juno-1}:supervisor.process_batch{chain=juno-1}:worker.channel{channel=channel::channel-1/wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8:juno-1->stargaze-1}:send_messages_and_wait_commit{chain=stargaze-1 tracking_id=ChannelOpenConfirm}:send_tx_with_account_sequence_retry{chain=stargaze-1 account.sequence=11}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call `send_tx_simulate` failed with status: status: Unknown, message: "account sequence mismatch, expected 12, got 11: incorrect account sequence [cosmos/cosmos-sdk@v0.45.16/x/auth/ante/sigverify.go:265] With gas wanted: '100000000000' and gas used: '64323' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "380"} }
2024-06-17T09:28:38.989512Z  WARN ThreadId(07) worker.batch{chain=juno-1}:supervisor.handle_batch{chain=juno-1}:supervisor.process_batch{chain=juno-1}:worker.channel{channel=channel::channel-1/wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8:juno-1->stargaze-1}:send_messages_and_wait_commit{chain=stargaze-1 tracking_id=ChannelOpenConfirm}:send_tx_with_account_sequence_retry{chain=stargaze-1 account.sequence=11}: failed to estimate gas because of a mismatched account sequence number, refreshing account sequence number and retrying once error=gRPC call `send_tx_simulate` failed with status: status: Unknown, message: "account sequence mismatch, expected 12, got 11: incorrect account sequence [cosmos/cosmos-sdk@v0.45.16/x/auth/ante/sigverify.go:265] With gas wanted: '100000000000' and gas used: '64323' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "380"} }
2024-06-17T09:28:38.990954Z  INFO ThreadId(07) worker.batch{chain=juno-1}:supervisor.handle_batch{chain=juno-1}:supervisor.process_batch{chain=juno-1}:worker.channel{channel=channel::channel-1/wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8:juno-1->stargaze-1}:send_messages_and_wait_commit{chain=stargaze-1 tracking_id=ChannelOpenConfirm}:send_tx_with_account_sequence_retry{chain=stargaze-1 account.sequence=11}: refreshed account sequence number old=11 new=12
2024-06-17T09:28:39.598115Z  INFO ThreadId(392) worker.batch{chain=juno-1}:supervisor.handle_batch{chain=juno-1}:supervisor.process_batch{chain=juno-1}:worker.channel{channel=channel::channel-1/wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8:juno-1->stargaze-1}: 🎊  stargaze-1 => OpenConfirmChannel(OpenConfirm { port_id: wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3, channel_id: channel-1, connection_id: connection-0, counterparty_port_id: wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8, counterparty_channel_id: channel-1 }) at height 1-381
2024-06-17T09:28:39.598143Z  INFO ThreadId(392) worker.batch{chain=juno-1}:supervisor.handle_batch{chain=juno-1}:supervisor.process_batch{chain=juno-1}:worker.channel{channel=channel::channel-1/wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8:juno-1->stargaze-1}: channel handshake step completed with events: OpenConfirmChannel(OpenConfirm { port_id: wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3, channel_id: channel-1, connection_id: connection-0, counterparty_port_id: wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8, counterparty_channel_id: channel-1 })
2024-06-17T09:28:40.593305Z  INFO ThreadId(08) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}:foreign_client.detect_misbehaviour{client=juno-1->stargaze-1:07-tendermint-0}: No evidence of misbehavior detected for chain juno-1
2024-06-17T09:28:40.693419Z  INFO ThreadId(10) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}: client is valid
2024-06-17T09:29:27.045256Z  INFO ThreadId(12) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.refresh{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.refresh{client=stargaze-1->juno-1:07-tendermint-0}: client needs to be refreshed elapsed=50.62723062s refresh_window=50s
2024-06-17T09:29:27.065334Z ERROR ThreadId(08) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.refresh{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.refresh{client=stargaze-1->juno-1:07-tendermint-0}:foreign_client.build_update_client_and_send{client=stargaze-1->juno-1:07-tendermint-0 target_query_height=latest height}:send_messages_and_wait_commit{chain=juno-1 tracking_id=update client}:send_tx_with_account_sequence_retry{chain=juno-1 account.sequence=11}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call `send_tx_simulate` failed with status: status: Unknown, message: "account sequence mismatch, expected 12, got 11: incorrect account sequence [cosmos/cosmos-sdk@v0.47.5/x/auth/ante/sigverify.go:269] With gas wanted: '100000000000' and gas used: '42967' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "180"} }
2024-06-17T09:29:27.065375Z  WARN ThreadId(08) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.refresh{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.refresh{client=stargaze-1->juno-1:07-tendermint-0}:foreign_client.build_update_client_and_send{client=stargaze-1->juno-1:07-tendermint-0 target_query_height=latest height}:send_messages_and_wait_commit{chain=juno-1 tracking_id=update client}:send_tx_with_account_sequence_retry{chain=juno-1 account.sequence=11}: failed to estimate gas because of a mismatched account sequence number, refreshing account sequence number and retrying once error=gRPC call `send_tx_simulate` failed with status: status: Unknown, message: "account sequence mismatch, expected 12, got 11: incorrect account sequence [cosmos/cosmos-sdk@v0.47.5/x/auth/ante/sigverify.go:269] With gas wanted: '100000000000' and gas used: '42967' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "180"} }
2024-06-17T09:29:27.066987Z  INFO ThreadId(08) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.refresh{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.refresh{client=stargaze-1->juno-1:07-tendermint-0}:foreign_client.build_update_client_and_send{client=stargaze-1->juno-1:07-tendermint-0 target_query_height=latest height}:send_messages_and_wait_commit{chain=juno-1 tracking_id=update client}:send_tx_with_account_sequence_retry{chain=juno-1 account.sequence=11}: refreshed account sequence number old=11 new=12
2024-06-17T09:29:29.477125Z  INFO ThreadId(09) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.refresh{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.refresh{client=juno-1->stargaze-1:07-tendermint-0}: client needs to be refreshed elapsed=50.432316803s refresh_window=50s
2024-06-17T09:29:29.498833Z ERROR ThreadId(07) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.refresh{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.refresh{client=juno-1->stargaze-1:07-tendermint-0}:foreign_client.build_update_client_and_send{client=juno-1->stargaze-1:07-tendermint-0 target_query_height=latest height}:send_messages_and_wait_commit{chain=stargaze-1 tracking_id=update client}:send_tx_with_account_sequence_retry{chain=stargaze-1 account.sequence=12}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call `send_tx_simulate` failed with status: status: Unknown, message: "account sequence mismatch, expected 13, got 12: incorrect account sequence [cosmos/cosmos-sdk@v0.45.16/x/auth/ante/sigverify.go:265] With gas wanted: '100000000000' and gas used: '54163' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "442"} }
2024-06-17T09:29:29.498873Z  WARN ThreadId(07) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.refresh{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.refresh{client=juno-1->stargaze-1:07-tendermint-0}:foreign_client.build_update_client_and_send{client=juno-1->stargaze-1:07-tendermint-0 target_query_height=latest height}:send_messages_and_wait_commit{chain=stargaze-1 tracking_id=update client}:send_tx_with_account_sequence_retry{chain=stargaze-1 account.sequence=12}: failed to estimate gas because of a mismatched account sequence number, refreshing account sequence number and retrying once error=gRPC call `send_tx_simulate` failed with status: status: Unknown, message: "account sequence mismatch, expected 13, got 12: incorrect account sequence [cosmos/cosmos-sdk@v0.45.16/x/auth/ante/sigverify.go:265] With gas wanted: '100000000000' and gas used: '54163' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "442"} }
2024-06-17T09:29:29.500496Z  INFO ThreadId(07) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.refresh{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.refresh{client=juno-1->stargaze-1:07-tendermint-0}:foreign_client.build_update_client_and_send{client=juno-1->stargaze-1:07-tendermint-0 target_query_height=latest height}:send_messages_and_wait_commit{chain=stargaze-1 tracking_id=update client}:send_tx_with_account_sequence_retry{chain=stargaze-1 account.sequence=12}: refreshed account sequence number old=12 new=13
2024-06-17T09:29:30.417736Z  INFO ThreadId(07) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=stargaze-1->juno-1:07-tendermint-0}:foreign_client.detect_misbehaviour{client=stargaze-1->juno-1:07-tendermint-0}: No evidence of misbehavior detected for chain stargaze-1
2024-06-17T09:29:30.517844Z  INFO ThreadId(13) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=stargaze-1->juno-1:07-tendermint-0}: client is valid
2024-06-17T09:29:31.309352Z  INFO ThreadId(08) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}:foreign_client.detect_misbehaviour{client=juno-1->stargaze-1:07-tendermint-0}: No evidence of misbehavior detected for chain juno-1
2024-06-17T09:29:31.409442Z  INFO ThreadId(10) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}: client is valid

Logs of hermes start when running from inside the pods to relay 1 packet: 

2024-06-17T09:34:06.661677Z  INFO ThreadId(01) using default configuration from '/root/.hermes/config.toml'
2024-06-17T09:34:06.662634Z  INFO ThreadId(01) running Hermes v1.7.4
2024-06-17T09:34:06.663884Z  INFO ThreadId(04) telemetry service running, exposing metrics at http://0.0.0.0:3001/metrics
2024-06-17T09:34:06.663936Z  INFO ThreadId(05) REST service running, exposing REST API at http://0.0.0.0:3000
thread '<unnamed>' panicked at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/server/server.rs:79:13:
error binding to 0.0.0.0:3001: error creating server listener: Address already in use (os error 98)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-0.14.27/src/server/server.rs:79:13:
error binding to 0.0.0.0:3000: error creating server listener: Address already in use (os error 98)
2024-06-17T09:34:06.664161Z ERROR ThreadId(04) telemetry service crashed with errror: task 2 panicked
2024-06-17T09:34:06.664163Z ERROR ThreadId(05) REST service crashed with errror: task 3 panicked
2024-06-17T09:34:06.670056Z  WARN ThreadId(07) health_check{chain=stargaze-1}: Chain 'stargaze-1' has no minimum gas price value configured for denomination 'ustars'. This is usually a sign of misconfiguration, please check your chain and relayer configurations
2024-06-17T09:34:06.675084Z  INFO ThreadId(07) health_check{chain=stargaze-1}: Updated `max_block_time` using /genesis endpoint. Old value: `30s`, new value: `30s`
2024-06-17T09:34:06.675112Z  INFO ThreadId(01) health_check{chain=stargaze-1}: chain is healthy
2024-06-17T09:34:06.679765Z  WARN ThreadId(08) health_check{chain=juno-1}: Chain 'juno-1' has no minimum gas price value configured for denomination 'ujuno'. This is usually a sign of misconfiguration, please check your chain and relayer configurations
2024-06-17T09:34:06.685488Z  INFO ThreadId(08) health_check{chain=juno-1}: Updated `max_block_time` using /genesis endpoint. Old value: `30s`, new value: `30s`
2024-06-17T09:34:06.685517Z  INFO ThreadId(01) health_check{chain=juno-1}: chain is healthy
2024-06-17T09:34:06.686137Z  INFO ThreadId(01) scan.chain{chain=stargaze-1}: scanning chain...
2024-06-17T09:34:06.686166Z  INFO ThreadId(01) scan.chain{chain=stargaze-1}: scanning chain for all clients, connections and channels
2024-06-17T09:34:06.686171Z  INFO ThreadId(01) scan.chain{chain=stargaze-1}: scanning all clients...
2024-06-17T09:34:06.687653Z  INFO ThreadId(01) scan.chain{chain=stargaze-1}:scan.client{client=07-tendermint-0}: scanning client...
2024-06-17T09:34:06.690129Z  INFO ThreadId(01) scan.chain{chain=stargaze-1}:scan.client{client=07-tendermint-0}:scan.connection{connection=connection-0}: scanning connection...
2024-06-17T09:34:06.694763Z  INFO ThreadId(01) scan.chain{chain=juno-1}: scanning chain...
2024-06-17T09:34:06.694798Z  INFO ThreadId(01) scan.chain{chain=juno-1}: scanning chain for all clients, connections and channels
2024-06-17T09:34:06.694801Z  INFO ThreadId(01) scan.chain{chain=juno-1}: scanning all clients...
2024-06-17T09:34:06.696507Z  WARN ThreadId(08) scan.chain{chain=juno-1}: encountered unsupported client type `/ibc.lightclients.localhost.v2.ClientState` while scanning client `09-localhost`, skipping the client
2024-06-17T09:34:06.696532Z  INFO ThreadId(01) scan.chain{chain=juno-1}:scan.client{client=07-tendermint-0}: scanning client...
2024-06-17T09:34:06.697857Z  INFO ThreadId(01) scan.chain{chain=juno-1}:scan.client{client=07-tendermint-0}:scan.connection{connection=connection-0}: scanning connection...
2024-06-17T09:34:06.700950Z  INFO ThreadId(01) scanned chains:
2024-06-17T09:34:06.700957Z  INFO ThreadId(01) # Chain: stargaze-1
  - Client: 07-tendermint-0
    * Connection: connection-0
      | State: OPEN
      | Counterparty state: OPEN
      + Channel: channel-0
        | Port: transfer
        | State: OPEN
        | Counterparty: channel-0
      + Channel: channel-1
        | Port: wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3
        | State: OPEN
        | Counterparty: channel-1
      + Channel: channel-2
        | Port: wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3
        | State: TRYOPEN
        | Counterparty: channel-1
# Chain: juno-1
  - Client: 07-tendermint-0
    * Connection: connection-0
      | State: OPEN
      | Counterparty state: OPEN
      + Channel: channel-0
        | Port: transfer
        | State: OPEN
        | Counterparty: channel-0
      + Channel: channel-1
        | Port: wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8
        | State: OPEN
        | Counterparty: channel-1

2024-06-17T09:34:06.700994Z  INFO ThreadId(01) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}: connection is OPEN, state on destination chain is OPEN chain=stargaze-1 connection=connection-0 counterparty_chain=juno-1
2024-06-17T09:34:06.701004Z  INFO ThreadId(01) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}: connection is already open, not spawning Connection worker chain=stargaze-1 connection=connection-0
2024-06-17T09:34:06.701020Z  INFO ThreadId(01) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}: no connection workers were spawn chain=stargaze-1 connection=connection-0
2024-06-17T09:34:06.701035Z  INFO ThreadId(01) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}: channel is OPEN, state on destination chain is OPEN chain=stargaze-1 counterparty_chain=juno-1 channel=channel-0
2024-06-17T09:34:06.703273Z  INFO ThreadId(01) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}: spawned client worker: client::juno-1->stargaze-1:07-tendermint-0
2024-06-17T09:34:06.705872Z  INFO ThreadId(01) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}: done spawning channel workers chain=stargaze-1 channel=channel-0
2024-06-17T09:34:06.705888Z  INFO ThreadId(01) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-1}: channel is OPEN, state on destination chain is OPEN chain=stargaze-1 counterparty_chain=juno-1 channel=channel-1
2024-06-17T09:34:06.709627Z  INFO ThreadId(01) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}: done spawning channel workers chain=stargaze-1 channel=channel-1
2024-06-17T09:34:06.709643Z  INFO ThreadId(01) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-2}: channel is TRYOPEN, state on destination chain is OPEN chain=stargaze-1 counterparty_chain=juno-1 channel=channel-2
2024-06-17T09:34:06.709656Z  INFO ThreadId(01) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}: no channel workers were spawned chain=stargaze-1 channel=channel-2
2024-06-17T09:34:06.709713Z  INFO ThreadId(01) spawn:chain{chain=stargaze-1}: spawning Wallet worker: wallet::stargaze-1
2024-06-17T09:34:06.709731Z  INFO ThreadId(01) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}: connection is OPEN, state on destination chain is OPEN chain=juno-1 connection=connection-0 counterparty_chain=stargaze-1
2024-06-17T09:34:06.709739Z  INFO ThreadId(01) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}: connection is already open, not spawning Connection worker chain=juno-1 connection=connection-0
2024-06-17T09:34:06.709746Z  INFO ThreadId(01) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}: no connection workers were spawn chain=juno-1 connection=connection-0
2024-06-17T09:34:06.709754Z  INFO ThreadId(01) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}: channel is OPEN, state on destination chain is OPEN chain=juno-1 counterparty_chain=stargaze-1 channel=channel-0
2024-06-17T09:34:06.712828Z  INFO ThreadId(01) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}: spawned client worker: client::stargaze-1->juno-1:07-tendermint-0
2024-06-17T09:34:06.715026Z  INFO ThreadId(01) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}: done spawning channel workers chain=juno-1 channel=channel-0
2024-06-17T09:34:06.715046Z  INFO ThreadId(01) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-1}: channel is OPEN, state on destination chain is OPEN chain=juno-1 counterparty_chain=stargaze-1 channel=channel-1
2024-06-17T09:34:06.717190Z  INFO ThreadId(01) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-1}: spawned packet worker: packet::channel-1/wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8:juno-1->stargaze-1
2024-06-17T09:34:06.717203Z  INFO ThreadId(01) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}: done spawning channel workers chain=juno-1 channel=channel-1
2024-06-17T09:34:06.717237Z  INFO ThreadId(01) spawn:chain{chain=juno-1}: spawning Wallet worker: wallet::juno-1
2024-06-17T09:34:06.721012Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:06.721058Z  INFO ThreadId(01) Hermes has started
2024-06-17T09:34:07.118312Z  INFO ThreadId(08) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}:foreign_client.detect_misbehaviour{client=juno-1->stargaze-1:07-tendermint-0}: No evidence of misbehavior detected for chain juno-1
2024-06-17T09:34:07.129687Z  INFO ThreadId(07) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=stargaze-1->juno-1:07-tendermint-0}:foreign_client.detect_misbehaviour{client=stargaze-1->juno-1:07-tendermint-0}: No evidence of misbehavior detected for chain stargaze-1
2024-06-17T09:34:07.221087Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:07.426607Z ERROR ThreadId(08) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}:foreign_client.detect_misbehaviour{client=juno-1->stargaze-1:07-tendermint-0}: could not detect misbehavior: light client builder error: invalid light block: not within trusting period: expires_at=2024-06-17T09:33:17.660355974Z now=2024-06-17T09:34:03.654065994Z
2024-06-17T09:34:07.426689Z  WARN ThreadId(10) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}: misbehaviour checking result: error raised while checking for misbehaviour evidence: failed to check misbehaviour for 07-tendermint-0 at consensus height 1-283: light client builder error: invalid light block: not within trusting period: expires_at=2024-06-17T09:33:17.660355974Z now=2024-06-17T09:34:03.654065994Z
2024-06-17T09:34:07.436939Z ERROR ThreadId(07) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=stargaze-1->juno-1:07-tendermint-0}:foreign_client.detect_misbehaviour{client=stargaze-1->juno-1:07-tendermint-0}: could not detect misbehavior: light client builder error: invalid light block: not within trusting period: expires_at=2024-06-17T09:33:19.435664381Z now=2024-06-17T09:34:06.146662642Z
2024-06-17T09:34:07.436998Z  WARN ThreadId(13) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=stargaze-1->juno-1:07-tendermint-0}: misbehaviour checking result: error raised while checking for misbehaviour evidence: failed to check misbehaviour for 07-tendermint-0 at consensus height 1-694: light client builder error: invalid light block: not within trusting period: expires_at=2024-06-17T09:33:19.435664381Z now=2024-06-17T09:34:06.146662642Z
2024-06-17T09:34:07.721168Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:08.221256Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:08.317697Z  INFO ThreadId(14) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-1}:worker.packet.cmd{src_chain=juno-1 src_port=wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8 src_channel=channel-1 dst_chain=stargaze-1}: clearing packets
2024-06-17T09:34:08.322787Z  INFO ThreadId(14) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-1}:worker.packet.cmd{src_chain=juno-1 src_port=wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8 src_channel=channel-1 dst_chain=stargaze-1}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 320 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 319 })}:schedule_recv_packet_and_timeout_msgs{query_height=1-319}: pulled packet data for 1 events out of 1 sequences: 3..=3; events.total=1 events.left=0
2024-06-17T09:34:08.351269Z  INFO ThreadId(14) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-1}:worker.packet.cmd{src_chain=juno-1 src_port=wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8 src_channel=channel-1 dst_chain=stargaze-1}:relay{odata=cleared/7328b735 ->Destination @1-319; len=1}: assembled batch of 2 message(s)
2024-06-17T09:34:08.363062Z  INFO ThreadId(14) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-1}:worker.packet.cmd{src_chain=juno-1 src_port=wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8 src_channel=channel-1 dst_chain=stargaze-1}:relay{odata=cleared/7328b735 ->Destination @1-319; len=1}: response(s): 1; Ok:ADF1EFEE65B82C68C7118EE06913755F1548094C0F6FF609E40881CC36981BD6 target_chain=stargaze-1
2024-06-17T09:34:08.363104Z  INFO ThreadId(14) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-1}:worker.packet.cmd{src_chain=juno-1 src_port=wasm.juno14hj2tavq8fpesdwxxcu44rty3hh90vhujrvcmstl4zr3txmfvw9skjuwg8 src_channel=channel-1 dst_chain=stargaze-1}:relay{odata=cleared/7328b735 ->Destination @1-319; len=1}: submitted
2024-06-17T09:34:08.721345Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:09.120248Z  INFO ThreadId(88) worker.batch{chain=stargaze-1}:supervisor.handle_batch{chain=stargaze-1}:supervisor.process_batch{chain=stargaze-1}:worker.packet.cmd{src_chain=stargaze-1 src_port=wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3 src_channel=channel-1 dst_chain=juno-1}: clearing packets
2024-06-17T09:34:09.129553Z  INFO ThreadId(88) worker.batch{chain=stargaze-1}:supervisor.handle_batch{chain=stargaze-1}:supervisor.process_batch{chain=stargaze-1}:worker.packet.cmd{src_chain=stargaze-1 src_port=wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3 src_channel=channel-1 dst_chain=juno-1}:schedule_packet_clearing{height=Some(Height { revision: 1, height: 784 })}:relay_pending_packets{height=Some(Height { revision: 1, height: 783 })}:build_packet_ack_msgs{query_height=1-783}: pulled packet data for 0 events out of 1 sequences: 3..=3; events.total=1 events.left=0
2024-06-17T09:34:09.221433Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:09.436297Z  INFO ThreadId(08) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}:foreign_client.detect_misbehaviour{client=juno-1->stargaze-1:07-tendermint-0}: No evidence of misbehavior detected for chain juno-1
2024-06-17T09:34:09.536442Z  INFO ThreadId(10) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}: client is valid
2024-06-17T09:34:09.556253Z  INFO ThreadId(88) worker.batch{chain=stargaze-1}:supervisor.handle_batch{chain=stargaze-1}:supervisor.process_batch{chain=stargaze-1}:worker.packet.cmd{src_chain=stargaze-1 src_port=wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3 src_channel=channel-1 dst_chain=juno-1}:relay{odata=a29fbf05 ->Destination @1-784; len=1}: assembled batch of 2 message(s)
2024-06-17T09:34:09.563571Z  INFO ThreadId(88) worker.batch{chain=stargaze-1}:supervisor.handle_batch{chain=stargaze-1}:supervisor.process_batch{chain=stargaze-1}:worker.packet.cmd{src_chain=stargaze-1 src_port=wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3 src_channel=channel-1 dst_chain=juno-1}:relay{odata=a29fbf05 ->Destination @1-784; len=1}: response(s): 1; Ok:25443079680233D28079E62817527F88945AB369984A110E720D0F5757EED4AA target_chain=juno-1
2024-06-17T09:34:09.563601Z  INFO ThreadId(88) worker.batch{chain=stargaze-1}:supervisor.handle_batch{chain=stargaze-1}:supervisor.process_batch{chain=stargaze-1}:worker.packet.cmd{src_chain=stargaze-1 src_port=wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3 src_channel=channel-1 dst_chain=juno-1}:relay{odata=a29fbf05 ->Destination @1-784; len=1}: submitted
2024-06-17T09:34:09.721524Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:10.221617Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:10.647652Z  INFO ThreadId(07) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=stargaze-1->juno-1:07-tendermint-0}:foreign_client.detect_misbehaviour{client=stargaze-1->juno-1:07-tendermint-0}: No evidence of misbehavior detected for chain stargaze-1
2024-06-17T09:34:10.721707Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:10.747773Z  INFO ThreadId(13) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=stargaze-1->juno-1:07-tendermint-0}: client is valid
2024-06-17T09:34:11.221798Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:11.721893Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:12.221988Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:12.722078Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:13.222170Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:13.722261Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:14.222356Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:14.722454Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:15.222548Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:15.722643Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:16.222738Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:16.722829Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:17.222924Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:17.723014Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:18.223107Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:18.723193Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:19.223283Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:19.723375Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:20.223470Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:20.723564Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:21.223652Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:21.723740Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:22.167877Z  INFO ThreadId(88) worker.batch{chain=stargaze-1}:supervisor.handle_batch{chain=stargaze-1}:supervisor.process_batch{chain=stargaze-1}:worker.packet.cmd{src_chain=stargaze-1 src_port=wasm.stars1wug8sewp6cedgkmrmvhl3lf3tulagm9hnvy8p0rppz9yjw0g4wtqmpd9x3 src_channel=channel-1 dst_chain=juno-1}: clearing packets
2024-06-17T09:34:22.223826Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:22.723915Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:23.224006Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:23.724095Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:24.224188Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:24.724279Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:25.224370Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:25.724464Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:26.224557Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:26.724665Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:27.224755Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:27.724847Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:28.224942Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:28.725036Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:29.225132Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:29.725229Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:30.225320Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:30.725413Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:31.225502Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:31.725590Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:32.225681Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:32.725766Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:33.225850Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:33.725939Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:34.226032Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:34.726125Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:35.226219Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:35.726311Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:36.226406Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:36.726497Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:37.226590Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:37.726681Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:38.226775Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:38.726865Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:39.226960Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:39.727054Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:40.227145Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:40.727234Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:41.227322Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:41.727409Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:42.227498Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:42.727594Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:43.227694Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:43.727790Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:44.227884Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:44.727976Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:45.228069Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:45.728162Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:46.228259Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:46.728357Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:47.228457Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:47.728556Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:48.228651Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:48.728752Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:49.228846Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:49.728939Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:50.229033Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:50.729121Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:51.229215Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:51.729305Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:52.229354Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:52.729446Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:53.229536Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:53.729627Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:54.229718Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:54.729806Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:55.229896Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:55.729988Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:56.230080Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:56.730170Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:57.230268Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:57.730361Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:58.230452Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:58.730540Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:58.773111Z  INFO ThreadId(09) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.refresh{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.refresh{client=juno-1->stargaze-1:07-tendermint-0}: client needs to be refreshed elapsed=50.413932923s refresh_window=50s
2024-06-17T09:34:59.230630Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:34:59.730717Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:35:00.151465Z  INFO ThreadId(08) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}:foreign_client.detect_misbehaviour{client=juno-1->stargaze-1:07-tendermint-0}: No evidence of misbehavior detected for chain juno-1
2024-06-17T09:35:00.230805Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:35:00.251597Z  INFO ThreadId(10) spawn:chain{chain=stargaze-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=juno-1 dst_chain=stargaze-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=juno-1->stargaze-1:07-tendermint-0}: client is valid
2024-06-17T09:35:00.730891Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:35:00.778941Z  INFO ThreadId(12) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.refresh{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.refresh{client=stargaze-1->juno-1:07-tendermint-0}: client needs to be refreshed elapsed=50.596649272s refresh_window=50s
2024-06-17T09:35:01.230984Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:35:01.731078Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:35:02.231170Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:35:02.731267Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:35:03.164386Z  INFO ThreadId(07) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=stargaze-1->juno-1:07-tendermint-0}:foreign_client.detect_misbehaviour{client=stargaze-1->juno-1:07-tendermint-0}: No evidence of misbehavior detected for chain stargaze-1
2024-06-17T09:35:03.231360Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:35:03.264520Z  INFO ThreadId(13) spawn:chain{chain=juno-1}:client{client=07-tendermint-0}:connection{connection=connection-0}:channel{channel=channel-0}:worker.client.misbehaviour{client=07-tendermint-0 src_chain=stargaze-1 dst_chain=juno-1}:foreign_client.detect_misbehaviour_and_submit_evidence{client=stargaze-1->juno-1:07-tendermint-0}: client is valid
2024-06-17T09:35:03.731447Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:35:04.231539Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:35:04.731627Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:35:05.231718Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:35:05.731815Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel
2024-06-17T09:35:06.231909Z ERROR ThreadId(23) rest: error while waiting for requests: receiving on an empty and disconnected channel

Thank you in advance !

Anmol1696 commented 1 week ago

is there a way i can simulate the txn that is supposed to be relayed? Bash or JS?

Kayanski commented 1 week ago

Hello ! I'm using RUST for the simulations : Here is the script that hangs (https://github.com/AbstractSDK/polytone/blob/starship-failing/scripts/src/bin/starship-transfer.rs) (cargo run --bin starship) Here is the script that works (https://github.com/AbstractSDK/polytone/blob/starship-failing/scripts/src/bin/starship.rs) (cargo run --bin starship-transfer)

Anmol1696 commented 1 week ago

Is it possible to have a bash of a cosmjs script for the same? I dont think i can debug with Starship in rust at this point. If we are able to reproduce it, should be possible.

Also on a side note, even for wasm to wasm contract on mesh-security, i do have some issues with relaying packets sometimes, which happenes with some dealy.

What i want to know:

Note

Maybe we should try to first use the most latest version of the chains both juno and stargaze as well as hermes. Can you tell me what versions are the chains are at? so we can do it directly.