Error in three node payment

[Srtake]

Description

Hello, I tried to follow the example Interledger with Ethereum and XRP On-Ledger Settlement and configure a three node network. The only difference between my configuration and the tutorial is that I changed XRP ledger of Bob and Charlie to a different Ethereum testnet due to my requirement.

The first two payments were good, but started from the third payment I got timeout errors. Here are what my logs say:

2021-03-18T08:02:27.918197279+00:00 DEBUG interledger_stream::client: Sending packet 894 with amount: 1 and encrypted STREAM packet: StreamPacket { sequence: 894, ilp_packet_type: Prepare, prepare_amount: 0, frames: [ StreamMoneyFrame { stream_id: 1, shares: 1 }, ConnectionNewAddressFrame { source_account: example.alice } ] }
2021-03-18T08:02:27.927173296+00:00 ERROR api: interledger_stream::crypto: Error decrypting Unspecified
2021-03-18T08:02:27.929382858+00:00 DEBUG api: interledger_stream::server: Unable to parse data, rejecting Prepare packet
2021-03-18T08:02:27.953913679+00:00  INFO api:{reject.code=T04 reject.message= reject.triggered_by=example.bob}: interledger-node: result="reject"
2021-03-18T08:02:27.954375417+00:00  WARN interledger_stream::client: Unable to parse STREAM packet from response data for sequence 894
2021-03-18T08:02:27.954422919+00:00 DEBUG interledger_stream::congestion: Rejected packet with T04 error. Amount in flight was: 1, decreasing max in flight to: 1
2021-03-18T08:02:27.954446327+00:00 DEBUG interledger_stream::client: Prepare 894 with amount 1 was rejected with code: T04 (500 left to send)

There seems to be some issues with crypto (maybe congestion?) module which make every payment packet return a T04 error. And I don't know if this helps, I got the same result when I was trying Interledger.js several days ago. It never happens when there are only two nodes (i.e. this example)

Configuration

Redis server:

redis-server --port 6379 &> ~/ilp-logs/redis-a-node.log &
redis-server --port 6380 &> ~/ilp-logs/redis-a-se-rinkeby.log &
redis-server --port 6381 &> ~/ilp-logs/redis-b-node.log &
redis-server --port 6382 &> ~/ilp-logs/redis-b-se-rinkeby.log &
redis-server --port 6383 &> ~/ilp-logs/redis-b-se-ropsten.log &
redis-server --port 6384 &> ~/ilp-logs/redis-c-node.log &
redis-server --port 6385 &> ~/ilp-logs/redis-c-se-ropsten.log &

Settlement engines:

# Start Alice's settlement engine (Rinkeby)
./ilp-settlement-ethereum \
--private_key {my_private_key} \
--confirmations 0 \
--poll_frequency 1000 \
--ethereum_url {my_provider} \
--connector_url http://127.0.0.1:7771 \
--redis_url redis://127.0.0.1:6380/ \
--asset_scale 6 \
--settlement_api_bind_address 127.0.0.1:3000 \
&> ~/ilp-logs/node-alice-settlement-engine-eth.log &

# Start Bob's settlement engine (Rinkeby)
./ilp-settlement-ethereum \
--private_key {my_private_key} \
--confirmations 0 \
--poll_frequency 1000 \
--ethereum_url {my_provider} \
--connector_url http://127.0.0.1:8771 \
--redis_url redis://127.0.0.1:6382/ \
--asset_scale 6 \
--settlement_api_bind_address 127.0.0.1:3001 \
&> ~/ilp-logs/node-bob-settlement-engine-rinkeby.log &

# Start Bob's settlement engine (Ropsten)
./ilp-settlement-ethereum \
--private_key {my_private_key} \
--confirmations 0 \
--poll_frequency 1000 \
--ethereum_url {my_provider} \
--connector_url http://127.0.0.1:8771 \
--redis_url redis://127.0.0.1:6383/ \
--asset_scale 6 \
--settlement_api_bind_address 127.0.0.1:3002 \
&> ~/ilp-logs/node-bob-settlement-engine-ropsten.log &

# Start Charlie's settlement engine (Ropsten)
./ilp-settlement-ethereum \
--private_key {my_private_key} \
--confirmations 0 \
--poll_frequency 1000 \
--ethereum_url {my_provider} \
--connector_url http://127.0.0.1:9771 \
--redis_url redis://127.0.0.1:6385/ \
--asset_scale 6 \
--settlement_api_bind_address 127.0.0.1:3003 \
&> ~/ilp-logs/node-charlie-settlement-engine-ropsten.log &

Nodes:

# Start Alice's node

alias ilp-node="cargo run --quiet --bin ilp-node --"
alias ilp-cli="cargo run --quiet --bin ilp-cli --"

ilp-node \
--ilp_address example.alice \
--secret_seed 8852500887504328225458511465394229327394647958135038836332350604 \
--admin_auth_token hi_alice \
--redis_url redis://127.0.0.1:6379/ \
--http_bind_address 127.0.0.1:7770 \
--settlement_api_bind_address 127.0.0.1:7771 \
--exchange_rate.provider CoinCap \
&> ~/ilp-logs/node-alice.log &

ilp-node \
--ilp_address example.bob \
--secret_seed 1604966725982139900555208458637022875563691455429373719368053354 \
--admin_auth_token hi_bob \
--redis_url redis://127.0.0.1:6381/ \
--http_bind_address 127.0.0.1:8770 \
--settlement_api_bind_address 127.0.0.1:8771 \
--exchange_rate.provider CoinCap \
&> ~/ilp-logs/node-bob.log &

ilp-node \
--secret_seed 1232362131122139900555208458637022875563691455429373719368053354 \
--admin_auth_token hi_charlie \
--redis_url redis://127.0.0.1:6384/ \
--http_bind_address 127.0.0.1:9770 \
--settlement_api_bind_address 127.0.0.1:9771 \
--exchange_rate.provider CoinCap \
&> ~/ilp-logs/node-charlie.log &

Accounts:

export ILP_CLI_API_AUTH=hi_alice
ilp-cli accounts create alice \
    --ilp-address example.alice \
    --asset-code ETH \
    --asset-scale 6 \
    --max-packet-amount 100 \
    --ilp-over-http-incoming-token alice_password \
    --settle-to 0 &> ~/ilp-logs/account-alice-alice.log
ilp-cli accounts create bob \
    --ilp-address example.bob \
    --asset-code ETH \
    --asset-scale 6 \
    --max-packet-amount 100 \
    --settlement-engine-url http://localhost:3000 \
    --ilp-over-http-incoming-token bob_password \
    --ilp-over-http-outgoing-token alice_password \
    --ilp-over-http-url http://localhost:8770/accounts/alice/ilp \
    --settle-threshold 500 \
    --min-balance -1000 \
    --settle-to 0 \
    --routing-relation Peer &> ~/ilp-logs/account-alice-bob.log &
ilp-cli --node http://localhost:8770 accounts create alice \
    --auth hi_bob \
    --ilp-address example.alice \
    --asset-code ETH \
    --asset-scale 6 \
    --max-packet-amount 100 \
    --settlement-engine-url http://localhost:3001 \
    --ilp-over-http-incoming-token alice_password \
    --ilp-over-http-outgoing-token bob_password \
    --ilp-over-http-url http://localhost:7770/accounts/bob/ilp \
    --settle-threshold 500 \
    --min-balance -1000 \
    --settle-to 0 \
    --routing-relation Peer &> ~/ilp-logs/account-bob-alice.log
ilp-cli --node http://localhost:8770 accounts create charlie \
    --auth hi_bob \
    --asset-code ETH \
    --asset-scale 6 \
    --settlement-engine-url http://localhost:3002 \
    --ilp-over-http-incoming-token charlie_password \
    --ilp-over-http-outgoing-token bob_other_password \
    --ilp-over-http-url http://localhost:9770/accounts/bob/ilp \
    --settle-threshold 0 \
    --settle-to -1000000 \
    --min-balance -10000000 \
    --routing-relation Child &> ~/ilp-logs/account-charlie-bob.log &
ilp-cli --node http://localhost:9770 accounts create charlie \
    --auth hi_charlie \
    --asset-code ETH \
    --asset-scale 6 \
    --ilp-over-http-incoming-token charlie_password \
    --settle-to 0 &> ~/ilp-logs/account-bob-bob.log
ilp-cli --node http://localhost:9770 accounts create bob \
    --auth hi_charlie \
    --ilp-address example.bob \
    --asset-code ETH \
    --asset-scale 6 \
    --settlement-engine-url http://localhost:3003 \
    --ilp-over-http-incoming-token bob_other_password \
    --ilp-over-http-outgoing-token charlie_password \
    --ilp-over-http-url http://localhost:8770/accounts/charlie/ilp \
    --settle-threshold 200000 \
    --settle-to -1000000 \
    --min-balance -10000000 \
    --routing-relation Parent &> ~/ilp-logs/account-bob-charlie.log

And finally my payment command:

    --amount 500 \
    --to http://localhost:9770/accounts/charlie/spsp```

[kincaidoneil]

Hey! Here are the docs for the Ethereum settlement engine. Since you're using a different testnet, I'd recommend also passing the chain_id argument to set that correctly. (Otherwise the transactions won't be correctly signed.) Let me know if that helps!

[Srtake]

Hey! Here are the docs for the Ethereum settlement engine. Since you're using a different testnet, I'd recommend also passing the chain_id argument to set that correctly. (Otherwise the transactions won't be correctly signed.) Let me know if that helps!

I added the parameter chain_id in my configuration, and changed the roles of Bob and Charlie accounts to PEER (otherwise there would still be many T04 errors). After the operation, things are much better. Thank you!