Scenario balance sum flakiness

rakanalh commented 5 years ago

Intro

When running BF1, the balance sum sometimes fails:

https://github.com/raiden-network/raiden/blob/e34025fd39504da04697cc3a84f76b0b3086d0d1/raiden/tests/scenarios/bf1_basic_functionality.yaml#L231

scenario_player.exceptions.legacy.ScenarioAssertionError: Expected sum value "1995000000000000000" for channel fields "balance". Actual value: "1106000000000000000".

Investigation required:

Why does it fail even though all transfers have been successful?
Could it be because the payment followed a different route and paid different fees?

Dominik1999 commented 5 years ago

I can pick this up. I think this is due to some latency for the assertion. If all payments go through, the sum of all channels of node 0 must be exactly that.

rakanalh commented 5 years ago

The way to reproduce this issue is by making sure:

The PFS is connected to transport04 (this is currently the case as transport04 is the only active transport for testing)
Update BF1 to use only transport04 for each client node as well.
Run BF1 and wait until the transfers from 3 -> 0 finish.
Balance summation should fail.

Dominik1999 commented 5 years ago

Small things to add:

that happened with a lot of scenarios since we put the scenario player nodes and the transport server on servers on different locations (worldwide, e.g. Toronto and Bangladesh)
when I have seen this error locally, I was able to solve this with introducing wait before the assertion

LefterisJP commented 5 years ago

Update BF1 to use only transport04 for each client node as well.

That seems to already be the case right? https://github.com/raiden-network/raiden/blob/4bed4d7ba504f011a92bb331cd200240c213aa09/raiden/tests/scenarios/bf1_basic_functionality.yaml#L43-L52

Dominik1999 commented 5 years ago

yes

LefterisJP commented 5 years ago

In order to be able to reproduce and debug this I need releases for raiden, contracts and services that are on the same page and use same contracts. At the moment and since Friday afternoon that is not the case and running this scenario (and I guess any other) with latest develop/dev fails with:

[TokenNetworkRegistry]Address 0x62436bbadB629498107278486C2A367FfdFEb2f3 has wrong code.. Please update your Raiden installation

LefterisJP commented 5 years ago

Now I am blocked by the PFS not being able to connect to matrix and as such not being able to function.

This in turn makes Raiden fail to start ...

There is an error with the pathfinding service with address https://pfs-goerli-with-fee.services-dev.raiden.network. Error Message: Extr
a data: line 1 column 5 (char 4). Raiden will shut down.

Dominik1999 commented 5 years ago

now it should work again

Dominik1999 commented 5 years ago

but you might be blocked by the matrix presence bug anyway ...

LefterisJP commented 5 years ago

@Dominik1999 How does the presence bug manifest in a scenario? Route not found I guess? Because this is what I am getting when I run it now.

Dominik1999 commented 5 years ago

exactly

Dominik1999 commented 5 years ago

there must be a route, the scenario is designed like that. It can also manifest in an expired transfer since the Matrix bug is also affecting the local view of the nodes ... they start sending the transfer, but it gets stuck

LefterisJP commented 5 years ago

As per our talk with @rakanalh during standup I set all transports to server 01 instead of 04 and ran again.

Now the scenario failed in a different place:

https://github.com/raiden-network/raiden/blob/e34025fd39504da04697cc3a84f76b0b3086d0d1/raiden/tests/scenarios/bf1_basic_functionality.yaml#L126

scenario_player.exceptions.legacy.ScenarioAssertionError: Expected request_count 11 but got 10

Is this a new thing that I should investigate or related to what you guys are doing?

Dominik1999 commented 5 years ago

haha fuck, that is another bug. We had this before @palango can help there is already an issue for that

Dominik1999 commented 5 years ago

ok if it helps, I had this fail now

│      - [✔] Transfer (0:00:04.782794): {'from': 0, 'to': 3, 'amount': 1000000000000000, 'lock_timeout': 30}                                                                                                                                                                                                                                             │
│      - [✔] Transfer (0:00:05.730143): {'from': 0, 'to': 3, 'amount': 1000000000000000, 'lock_timeout': 30}                                                                                                                                                                                                                                             │
│      - [✔] Transfer (0:00:04.763107): {'from': 0, 'to': 3, 'amount': 1000000000000000, 'lock_timeout': 30}                                                                                                                                                                                                                                             │
│      - [✔] Transfer (0:00:04.827382): {'from': 0, 'to': 3, 'amount': 1000000000000000, 'lock_timeout': 30}                                                                                                                                                                                                                                             │
│   - [✗] Serial (0:01:40.033207) - Assert after 100 payments from 0 to 3                                                                                                                                                                                                                                                                                │
│      - [✔] Wait (0:01:40.000174): 100                                                                                                                                                                                                                                                                                                                  │
│      - [✗] AssertSum (0:00:00.025140): {'from': 0, 'balance_sum': 1905000000000000000}                                                                                                                                                                                                                                                                 │
│      - [ ] AssertSum: {'from': 3, 'balance_sum': 2090000000000000000}

Expected sum value \"1905000000000000000\" for channel fields \"balance\". Actual value: \"1904976900000000000\"

All transfers are green but the balance check fails even after 100s (so 31 transfers succeeded but the balance was not updated)

BF1 fail Assert Sum.zip

Dominik1999 commented 5 years ago

Now with a second try it failed with an even weirder value │scenario_player.exceptions.legacy.ScenarioAssertionError: Expected sum value "1995000000000000000" for channel fields "balance". Actual value: "1105000000000000000".

Bf1-fail- Assert Sum.zip

LefterisJP commented 5 years ago

Some thoughts after analyzing Dominik's logs

@Dominik1999 @rakanalh I think we see two different issues here.

I took a look at the logs given by Dominik in the comment just above this one here.

Looking at past logs and the issue description the error always seems to happen at this line.

Let's see what happens in this scenario.

The network layout looks like this:

0 <-> 1 <-> 2 <-> 3
^                 ^
|                 |
------->4<---------

The only payments going out from node 0 are:

5 payments of 1_000_000_000_000_000 from 0 to 2
100 payments of 1_000_000_000_000_000 from 0 to 3

The logs of 0 seem to agree with this. They go 105 PaymentSentSuccess events.

But the scenario logs fail. The error above is:

expected: 1_995_000_000_000_000_000 but got : 1_105_000_000_000_000_000

That's really weird since after sending 100 payments the balance should not go down.

The assertion uses the following scenario player task:

https://github.com/raiden-network/scenario-player/blob/7b64540a050b1e351c9c4dab78ca021bbaae583d/scenario_player/tasks/channels.py#L218-L245

which should iterate all channel states so we should see the sum of all balances for 0, both from channel 0-1 and channel 0-4.

Issue 1

Which brings me to issue 1. I think the scenario player has a bug in that assertion that it can sometimes fail and not take an entire channel into account.

It's very easy to confirm by taking the DB of node 0 from the logs above and checking the balances.

final_balance of 0 for channel_01: 1005000000000000000
final_balance of 0 for channel_04: 899978900000000000

1005000000000000000 should be familiar as this is the balance sum we get when the assertion failed.

From this I can deduce that the scenario player has a bug at getting the balance sum of all channels of a node. Somehow it can omit to take into account balances of all channels.

Issue 2

From above , even if you add both balances 1005000000000000000 and 899978900000000000 they add up to 1904978900000000000 and not 1999500000000000000 so there is a difference of 94521100000000000

That's 94_521_100_000_000_000 .. which is ~94x the amount transfered in a single transfer.

I suspect something is wrong there. But ... with all the changes in fees ... and fees schedules are the amounts set as expected in the scenario correct?

LefterisJP commented 5 years ago

Addendum

Hmm wait there is more. The scenarios' expected balances seem wrong to me at exactly the point of the failure

0 <-> 1 <-> 2 <-> 3
^                 ^
|                 |
------->4<---------

With the above network layout in mind let's look at the timeline of the scenario for Node 0.

T0 - Start

channel0_1:      1_000_000_000_000_000_000
channel0_4:      1_000_000_000_000_000_000
total_balance_0: 2_000_000_000_000_000_000

T1 - Make 10 transfers from 3 to 0

channel0_1:      1_010_000_000_000_000_000
channel0_4:      1_000_000_000_000_000_000
total_balance_0: 2_010_000_000_000_000_000

Scenario assertion for Node0's total balance at time T1 is here:

https://github.com/raiden-network/raiden/blob/88664b202291c4d594e7cebf55d91cd8f41cf862/raiden/tests/scenarios/bf1_basic_functionality.yaml#L108

T2 - Make payments from 0 to 2 after withdraw

channel0_1:      1_005_000_000_000_000_000
channel0_4:      1_000_000_000_000_000_000
total_balance_0: 2_005_000_000_000_000_000

Scenario assertion for Node0's total balance at time T2 is here:

https://github.com/raiden-network/raiden/blob/88664b202291c4d594e7cebf55d91cd8f41cf862/raiden/tests/scenarios/bf1_basic_functionality.yaml#L161

T3 - Make 100 payments from 0 to 3

channel0_1:      1_005_000_000_000_000_000
channel0_4:        900_000_000_000_000_000
total_balance_0: 1_905_000_000_000_000_000

Scenario assertion for Node0's total balance at time T3 is here:

https://github.com/raiden-network/raiden/blob/88664b202291c4d594e7cebf55d91cd8f41cf862/raiden/tests/scenarios/bf1_basic_functionality.yaml#L189

T4 - Make 10 payments from 0 to 3 after restart

channel0_1:      1_105_000_000_000_000_000
channel0_4:        890_000_000_000_000_000
total_balance_0: 1_995_000_000_000_000_000

Scenario assertion for Node0's total balance at time T4 is here:

https://github.com/raiden-network/raiden/blob/88664b202291c4d594e7cebf55d91cd8f41cf862/raiden/tests/scenarios/bf1_basic_functionality.yaml#L231

So far so good ...

The problem!!

The assertion for Node0's balance at T4 is what fails all the time. At a first glance it should be 1_995_000_000_000_000_000, but it seems the channel 0-4 is closed right before.

Sure enough

https://github.com/raiden-network/raiden/blob/88664b202291c4d594e7cebf55d91cd8f41cf862/raiden/tests/scenarios/bf1_basic_functionality.yaml#L208-L221

The channel 0-4 is closed before T4.

Which means the channel should not be counted in the balance calculation and only the channel 0-1 should be ... whose total balance should be 1_105_000_000_000_000_000 ... which is exactly what @Dominik1999's run above showed.

Sidenote

I am not sure if the transfers will pick the routes I picked for them during my scenario timeline creation. I made the assumption they would always go for the shortest route.

rakanalh commented 5 years ago

So if the channel is pending close (or closed, not sure), that means that the API called here:

https://github.com/raiden-network/scenario-player/blob/7b64540a050b1e351c9c4dab78ca021bbaae583d/scenario_player/tasks/channels.py#L163-L165

Would be called and the closed channel would be returned along with the the response?

If so, which is a more suitable solution?

Not return pending close or closed channels
Filtering those in the SP

LefterisJP commented 5 years ago

As discussed in the standup the SP should be testing what happens in the user's case. And since the users also see the closed channels this should be returned and also tested in the SP.

The question for last log of Dominik then becomes ... why was the closed channel not seen? Either an SP bug, or a timing issue in the scenario where there was a big pause inbetween and the channel ended up also being settled, in which case it did indeed get deleted and not returned in the channels list.

Dominik1999 commented 5 years ago

that might be related

…and the servers are in bangalore, singapore, toronto and amsterdam

I can locally reprocude the error when all nodes use transport03 (since the pfs goerli with fee is currently using that)

LefterisJP commented 5 years ago

All right I managed to reproduce this in a local scenario run. I found out exactly what is happening but not sure why.

What we know so far

As we have seen for far the scenario fails consistently here:

https://github.com/raiden-network/raiden/blob/e34025fd39504da04697cc3a84f76b0b3086d0d1/raiden/tests/scenarios/bf1_basic_functionality.yaml#L231

As we have seen above the big discrepancy in the scenario's node 0 balance is due to the channel 0-4 having been closed and settled. Either adjusting the scenario's expected balance or increasing the settlement timeout should fix this.

But then the balance should have been 1105000000000000000 and not 1105021100000000000. So where do the extra 21100000000000 come from?

My new run

Right after closing channel 4

Assertion Failure From 3 to 0:

Expected sum value \"1995000000000000000\" for channel fields \"balance\". Actual value: \"1105021100000000000\

0 <-> 1 <-> 2 <-> 3
^                 ^
|                 |
------->4<---------

Node 0: 0xFEbfb5a3756550e6F18Aa4B15370C8642482753D Node 1: 0xFCDdFC4AC4ea6b6940f687B9e7De7a26e08a06CC Node 2: 0xd8D69168C7b943bdEF6C8578922075872fDE90De Node 3: 0x45e3843891B72Df5DA0AC00dc42731699a133b86 Token network address: 0xDa1fBc048f503635950058953f5c60FC1F564ee6

Checking node 3's Database

All Node 3's EventPaymentSentSuccess events have amount 1000000000000000 so no discrepancies there.

Checking node 0's Database

Offending payment received is at channel0_1 and is here:

EventPaymentReceivedSuccess< token_network_address: 0xDa1fBc048f503635950058953f5c60FC1F564ee6 identifier: 15188145939394927688 amount: 1021100000000000 initiator: 0x45e3843891B72Df5DA0AC00dc42731699a133b86 >
EventUnlockClaimSuccess(identifier=15188145939394927688, secrethash=b'\x17\x84\x7f\xe46\xf0\xabpK\xe4\xdfsb\x1a\x90\xaa\xaa\x19K-\xf6\x1e\xefJ\x9b\xfd)2m}\xb9\x8a')

It's sent from Node 3 and the EventPaymentReceivedSuccess contains the offending amount 1021100000000000.

Checking node 3's Database again

The EventPaymentSentSuccess for the same payment identifier 15188145939394927688 has the correct amount.

EventPaymentSentSuccess< token_network_address: 0xDa1fBc048f503635950058953f5c60FC1F564ee6 identifier: 15188145939394927688 amount: 1000000000000000 target: 0xFEbfb5a3756550e6F18Aa4B15370C8642482753D secret: 0xf70a301e14adf7f7bb3980c14d9b38dde2d1a93fdbc94fcf9f74b9c74391d16f route: [0x45e3843891B72Df5DA0AC00dc42731699a133b86,0xd8D69168C7b943bdEF6C8578922075872fDE90De,0xFCDdFC4AC4ea6b6940f687B9e7De7a26e08a06CC,0xFEbfb5a3756550e6F18Aa4B15370C8642482753D] >

The route is 3 -> 2 -> 1 -> 0 as expected.

But then checking the SendLockedTransfer event with the same payment identifier we see the offending token amount in this event.

SendLockedTransfer(recipient=b'\xd8\xd6\x91h\xc7\xb9C\xbd\xefl\x85x\x92 u\x87/\xde\x90\xde', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\xda\x1f\xbc\x04\x8fP65\x95\x00X\x95?\\`\xfc\x1fVN\xe6', channel_identifier=1128), message_identifier=11254132672163043241, queue_identifier=QueueIdentifier(recipient=b'\xd8\xd6\x91h\xc7\xb9C\xbd\xefl\x85x\x92 u\x87/\xde\x90\xde', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\xda\x1f\xbc\x04\x8fP65\x95\x00X\x95?\\`\xfc\x1fVN\xe6', channel_identifier=1128)), transfer=LockedTransferUnsignedState(payment_identifier=15188145939394927688, token=b'b\x08<\x805=\xf7qBm \x9e\xf5xa\x9e\xe6\x8d\\z', lock=HashTimeLockState(amount=1021100000000000, expiration=1561231, secrethash=b'\x17\x84\x7f\xe46\xf0\xabpK\xe4\xdfsb\x1a\x90\xaa\xaa\x19K-\xf6\x1e\xefJ\x9b\xfd)2m}\xb9\x8a'), initiator=b"E\xe3\x848\x91\xb7-\xf5\xda\n\xc0\r\xc4'1i\x9a\x13;\x86", target=b'\xfe\xbf\xb5\xa3ueP\xe6\xf1\x8a\xa4\xb1Sp\xc8d$\x82u=', balance_proof=BalanceProofUnsignedState(nonce=154, transferred_amount=76000000000000000, locked_amount=1021100000000000, locksroot=b')\xe8\xaa\xb8\x83\xbc-.{1\nc\xd8`\x9e\xc1\xf0\x1d\xdd\xe2\x1a\xd1\x89V}\xda\x9c\xc7\xb8\xff\xa9T', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\xda\x1f\xbc\x04\x8fP65\x95\x00X\x95?\\`\xfc\x1fVN\xe6', channel_identifier=1128), balance_hash=HexBytes('0x62795b599ed639ac8c7ab1ffc72b858b80b0b43ab6ca6af87039b72afba111cd')), route_states=[RouteState (0xd8D69168C7b943bdEF6C8578922075872fDE90De -> 0xFCDdFC4AC4ea6b6940f687B9e7De7a26e08a06CC -> 0xFEbfb5a3756550e6F18Aa4B15370C8642482753D), channel_id: 1128, fee: 0]))

The route states here have no fees ... but if you check the state change that generated it:

ActionInitInitiator(transfer=TransferDescriptionWithSecretState(amount=1000000000000000, token_network_address=b'\xda\x1f\xbc\x04\x8fP65\x95\x00X\x95?\\`\xfc\x1fVN\xe6', target=b'\xfe\xbf\xb5\xa3ueP\xe6\xf1\x8a\xa4\xb1Sp\xc8d$\x82u=', secrethash=b'\x17\x84\x7f\xe46\xf0\xabpK\xe4\xdfsb\x1a\x90\xaa\xaa\x19K-\xf6\x1e\xefJ\x9b\xfd)2m}\xb9\x8a', lock_timeout=30), routes=[RouteState (0x45e3843891B72Df5DA0AC00dc42731699a133b86 -> 0xd8D69168C7b943bdEF6C8578922075872fDE90De -> 0xFCDdFC4AC4ea6b6940f687B9e7De7a26e08a06CC -> 0xFEbfb5a3756550e6F18Aa4B15370C8642482753D), channel_id: 1128, fee: 20000000000000, RouteState (0x45e3843891B72Df5DA0AC00dc42731699a133b86 -> 0x3641Cf7167fB3f53339f84A335C7EB7f0E3d42f7 -> 0x45e3843891B72Df5DA0AC00dc42731699a133b86 -> 0xd8D69168C7b943bdEF6C8578922075872fDE90De -> 0xFCDdFC4AC4ea6b6940f687B9e7De7a26e08a06CC -> 0xFEbfb5a3756550e6F18Aa4B15370C8642482753D), channel_id: 1130, fee: 20000000000000])

There they do have a fee of 20000000000000. Which along with a token amount of 1000000000000000 and the fee calculation formula 1000000000000000 +20000000000000 + int(ceil(20000000000000 * 0.03 + 1000000000000000 * 0.0005)) gives us 1021100000000000 which is our offending amount.

The remaining question

This estimated fee is put in the initial ActionInitInitiator state change after we query the PFS:

https://github.com/raiden-network/raiden/blob/f317455f82f70153f4b0f9ff6de72eef73824f79/raiden/routing.py#L82-L100

And it only seems to be a non zero number for the 99th out of the 100 payments of the "Make 100 payments from 3 to 0" task.

~~So now the question is why does the PFS return non-zero fees only for that one transfer?~~

Check the post below to see that the PFS query fails, and we fall back to internal routing for that particular transfer.

LefterisJP commented 5 years ago

Checking all PFSFeeUpdates sent from nodes

Continuing from the above findings I checked the PFSFeeUpdates of logs of all nodes of the scenario in case for some reason a node sent a non-zero fee update.

Node 0

{"message": "PFSFeeUpdate(signature=b'[\\x02\\x01\\xd0\\x94\\x8b\\xe2\\xb0?&<\\xc9\\x92\\xb7G\\x1f\\xb33\\x16\\x1d\\x0b\\x94<\\x87\\x1e09d\\xa7\\x96\\xff2~l\\xdf\\x07\\xa64\\x86A\\x124\\xf0\\xdb\\xc6\\xe1\\tl\\xf5\\xdc\\xad\\x7f\\x15\\x83`\\xf5\\x1e\\x08\\x0e.c\\x8b\\xc7$\\x1c', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), updating_participant=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 38, 10, 263785, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), our_state=NettingChannelEndState(address=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', contract_balance=0, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:38:10.265159"}
{"message": "PFSFeeUpdate(signature=b'@\\'F\\xdd\\xe6\\xf0\\xa2\\xc1\\xe9q\"\\xeb\\x9f\\xd78\\x81\\xc0\\xdb\\x04\\xffAlz\\x0f\\xfd\\x83\\x0b\\xe2\\xa7\\x88\\x12G\\x14\\x83\\xd3W\\x88]w\\xf2\\x19v\\xb2Ra\\xd0\\x03\\x99\\x19\\xbc\\xc9\\x92\\xee\\xd6\\xb0\\xfd\\x06\\xe6\\xb1\\x01\\xb2\\xfaI\\xe3\\x1b', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1132), updating_participant=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 41, 52, 392898, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1132), our_state=NettingChannelEndState(address=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'6A\\xcfqg\\xfb?S3\\x9f\\x84\\xa35\\xc7\\xeb\\x7f\\x0e=B\\xf7', contract_balance=0, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560860, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:41:52.394273"}
{"message": "PFSFeeUpdate(signature=b'R\\xbf\\x1bB\\xcf\\x8c\\xab\\xf1q\\xf7\\x93\\x0e\\x11\\xa7\\x95;\\xb0i-\\x98\\x1ee \\xe0\\xf9\\x8a\\x11%s\\xcd\\x1e\\xf4c2\\x17\\xd1\\x05\\xc7\\xcd\\xd0\\x10\\xb3H=\\xd2\\x92\\x01o3-\\xe8\\x8a\\x9bsr\\x18h\\xa7$\\xb8\\xc2F\\xb2 \\x1c', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1132), updating_participant=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 45, 10, 55973, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1132), our_state=NettingChannelEndState(address=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'6A\\xcfqg\\xfb?S3\\x9f\\x84\\xa35\\xc7\\xeb\\x7f\\x0e=B\\xf7', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560860, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:45:10.056949"}
{"message": "PFSFeeUpdate(signature=b'$\\xe5(\\xd6Z\\x9afs\\x96\\x89\\x00L\\xe5\\xea\\x0bR\\xee\\x8d\\xacB?\\x053\\xd0c\\xcf\\x16F+\\xc5\\x82\\xe1{\\xf6\\x08\\x13\\x194\\xe6\\xc6\\xff\\xd3\\x12%\\x91\\xe9I\\xae/#\\xb0\\xa6e\\xda>\\xda>\\xcfX\\x110\\xaf\\x94\\xda\\x1c', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), updating_participant=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 45, 10, 59309, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), our_state=NettingChannelEndState(address=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:45:10.060295"}
{"message": "PFSFeeUpdate(signature=b'VI\\x04\\xec\\xb0\\xd8a1\\x9e\\x88\\xbaMe\\xe8\\xeel\\xe9{\\xecv\\xd2WS\\x9a\\r7x-\\xda\\xb1\\xcb\\xf4H\\xb5}\\x94 \\x15\\xaa8\\xecy\\xeb\\xf5\\xf1l\\xad\\x96\\x91\\xe4k?\\xce7Xp7;\\xc8\\xb7\\xf5\\x9e\\xb5\\x1a\\x1c', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), updating_participant=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 11, 15, 51, 317159, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), our_state=NettingChannelEndState(address=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=BalanceProofUnsignedState(nonce=10, transferred_amount=5000000000000000, locked_amount=0, locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), balance_hash=HexBytes('0x783ad7fec10ccb442d179ddcea9edc5b2f7f435538a24a07a0c51760571c2257')), onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=10), partner_state=NettingChannelEndState(address=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', contract_balance=1100000000000000000, onchain_total_withdraw=0, balance_proof=BalanceProofSignedState< nonce: 20 transferred_amount: 10000000000000000 locked_amount: 0 locksroot: 0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470 message_hash: 0xa4151b94b8fd1708ca5bb8c7a34e1309c513029f42de7a9df272e896bdecf67d signature: 0x6a8ad55eea86d99d2463c20eb6b372259162c3a4c0ce08b69ea2f51e1f10d82204d546ab9f8280bd2d2d0728cf54b72bb7615c0a34d2515b48c673b88b7762151b sender: 0xFCDdFC4AC4ea6b6940f687B9e7De7a26e08a06CC canonical_identifier: CanonicalIdentifier(chain_identifier=5, token_network_address=0xDa1fBc048f503635950058953f5c60FC1F564ee6, channel_identifier=1129) balance_hash: 0xdac78515be87e45c9318c6924fb5c03b65eab7bdbd5967344d09e69a97c80d07 >, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=20), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 11:15:51.318327"}
{"message": "PFSFeeUpdate(signature=b'\\xd1\\xffA\\x02\\xcd\\xe4[\\xb7\\x9a\\xc6H\\x86\\x19\\x07\\x18\\xd4\\xe8\\x0e{\\xbf=\\xd5^Uy\\xfb{\\x86\\xfd\"\\x07X?\\xe0[\\x88,\\xdd\\x80\\x9d\\xd9\\x9e\\x102Y\\x1bx\\xa13]7\\xee\\x9dr\\xfbo\\xfdwun\\x91\\xd0\\xaeA\\x1b', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), updating_participant=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 11, 39, 5, 134059, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), our_state=NettingChannelEndState(address=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=BalanceProofUnsignedState(nonce=10, transferred_amount=5000000000000000, locked_amount=0, locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), balance_hash=HexBytes('0x783ad7fec10ccb442d179ddcea9edc5b2f7f435538a24a07a0c51760571c2257')), onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=10), partner_state=NettingChannelEndState(address=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', contract_balance=1100000000000000000, onchain_total_withdraw=0, balance_proof=BalanceProofSignedState< nonce: 20 transferred_amount: 10000000000000000 locked_amount: 0 locksroot: 0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470 message_hash: 0xa4151b94b8fd1708ca5bb8c7a34e1309c513029f42de7a9df272e896bdecf67d signature: 0x6a8ad55eea86d99d2463c20eb6b372259162c3a4c0ce08b69ea2f51e1f10d82204d546ab9f8280bd2d2d0728cf54b72bb7615c0a34d2515b48c673b88b7762151b sender: 0xFCDdFC4AC4ea6b6940f687B9e7De7a26e08a06CC canonical_identifier: CanonicalIdentifier(chain_identifier=5, token_network_address=0xDa1fBc048f503635950058953f5c60FC1F564ee6, channel_identifier=1129) balance_hash: 0xdac78515be87e45c9318c6924fb5c03b65eab7bdbd5967344d09e69a97c80d07 >, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=20), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 11:39:05.135114"}
{"message": "PFSFeeUpdate(signature=b'q\\x1f\\xe9?\\x92hk\\xb1;?\\xe7\\xcb\\xd2\\xa0\\x8d\\xad\\xcd\\xfc\\x89\\xb4\\xb0\\x8b\\x81\\x8f\\xbe\\xf6\\xc9\\xb7\\x1cF\\xf6\\xabw\\xfaZ\\xcd\\xd5\\xab\\xf5\\xd7\\xd1\\xdb\\xfb\\xdbJJ\\xb8&\\x8b\\x077\\x03@\\x10\\xbf\\x8e\\xf3\\x0e0\\r7\\x17\\x02\\xee\\x1c', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1132), updating_participant=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 11, 39, 5, 138159, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1132), our_state=NettingChannelEndState(address=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=BalanceProofUnsignedState(nonce=220, transferred_amount=110000000000000000, locked_amount=0, locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1132), balance_hash=HexBytes('0x7d640d09e47ad52ae76cf8b9b4fc68297a109ee5c097f1eb88ae83855ffb1b8e')), onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=220), partner_state=NettingChannelEndState(address=b'6A\\xcfqg\\xfb?S3\\x9f\\x84\\xa35\\xc7\\xeb\\x7f\\x0e=B\\xf7', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=BalanceProofSignedState< nonce: 20 transferred_amount: 10000000000000000 locked_amount: 0 locksroot: 0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470 message_hash: 0x7b7f2d989aa9d7f4470587ed5e7bcab42a7b87c743c3a98770b64f6ce524bd25 signature: 0x51272450777bd7ecf068559074294000b4ce5deb013836a11173c7f123364eb3012da57361ac9b9a15158f5a757c7168c7b9bc95105f36ca01267c5ecc58698b1c sender: 0x3641Cf7167fB3f53339f84A335C7EB7f0E3d42f7 canonical_identifier: CanonicalIdentifier(chain_identifier=5, token_network_address=0xDa1fBc048f503635950058953f5c60FC1F564ee6, channel_identifier=1132) balance_hash: 0xdac78515be87e45c9318c6924fb5c03b65eab7bdbd5967344d09e69a97c80d07 >, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=20), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560860, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 11:39:05.139161"}

Node 1

{"message": "PFSFeeUpdate(signature=b'\\xce\\xc9\\x89\\xee\\xee\\xff\\x98vr\\xc2\\xe8\\x11\\xc2\\xeeOb\\rC\\xb8\\xc7\\x88]H\\xc0\\xfb}\\x9e\\x18\\xdf\\xfb2\\xbe\\x01\\x80KfP\\x15\\x92\\x11\\xf0\\x9b\\xe8\\xc6.<4\\x06\\x96\\xcc\\xd6\\xec\\x0b\\x85\\xa6\\xdc\\x16\\xb7`\\xe5{\\xa186\\x1b', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), updating_participant=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 38, 9, 565775, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), our_state=NettingChannelEndState(address=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', contract_balance=0, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:38:09.567200"}
{"message": "PFSFeeUpdate(signature=b'\\xd2\\x85@E\\xa2\\xccZ\\xe9n\\x9a\\xff\\xdfg\\xff/\\xa8\\xcax\\x10\\xfe$\\xa28\\xbf}\\xfa?y\\xa3\\xe7Xx\\x1a\\x96\\xf9W\\xc6d1\\xeb\\xba\\x02\\xe5\\xd1\\xf4\\xf31T4\\xc5Ua<\\x18d\\xcf\\xd0\\x89\\xe1\\x11\"\\x07\\xbfN\\x1c', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1131), updating_participant=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 38, 9, 570363, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1131), our_state=NettingChannelEndState(address=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', contract_balance=0, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:38:09.571400"}
{"message": "PFSFeeUpdate(signature=b'\\x9e\\xa0\\xec\\xb5T\\xde.\\x81\\xc7\\xe3\\xee5L\\xabR\\x8cN\\x0e\\r<V\\xf4\\x16U\\xf2\\xff)\\x9ck\\xaa \\x182\\xcb\\xcb\\xdf{\\xfc\\x1fZw\\x9e\\xc7\\x06\\xf4\\x11\\xcd\\xd1\\x14\\xa3\\xcc\\xd0\\n\\x93\\xa1\\x88\\xd2\\xeeu\\x10\\xdd3\\x90\\xd1\\x1b', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), updating_participant=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 45, 9, 234208, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), our_state=NettingChannelEndState(address=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:45:09.235758"}
{"message": "PFSFeeUpdate(signature=b'\\xb3\\xea\\xabE\\x7f\\t\\x15H\\x16\\xa2\\r1\\x8dR\\xfd\\xdaf\\xf9>\\xc1.\\'9ga_u\\xb6h\\xec\\x9e\\xd3\\x0e\\x13\\x02&\\xd6V\\xe9o\\xc5\\xef\\x9d\\x07\\xe8\\xbb\\xb9L9^,E@\\xcf\\xb9H\\x12\\x03\\x8f`Lb\"\\x1d\\x1c', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1131), updating_participant=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 45, 9, 238492, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1131), our_state=NettingChannelEndState(address=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:45:09.239551"}
{"message": "PFSFeeUpdate(signature=b'=\\x02`\\xa4\\x1a\\xe1,&2\\xfd\\x067\\xf7:MD\\xacK\\\\\\x94V\\x80\\xdd{\\x12h\\x99\\x8a\\xf2\\xb3\\xd6\\x1a$\\x19\\xa1\\x05\\xafG\\x1c\\xb4\\x0fQ\\xfd\\x04\\xaak\\x8d\\xf0\\x86\\xc0\\x86\\xfe\\xccH\\x84\\xd3\\x87\\xea\\x02\\xaa\\xf97Fe\\x1b', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), updating_participant=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 11, 15, 51, 36661, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), our_state=NettingChannelEndState(address=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', contract_balance=1100000000000000000, onchain_total_withdraw=0, balance_proof=BalanceProofUnsignedState(nonce=20, transferred_amount=10000000000000000, locked_amount=0, locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1129), balance_hash=HexBytes('0xdac78515be87e45c9318c6924fb5c03b65eab7bdbd5967344d09e69a97c80d07')), onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=20), partner_state=NettingChannelEndState(address=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=BalanceProofSignedState< nonce: 10 transferred_amount: 5000000000000000 locked_amount: 0 locksroot: 0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470 message_hash: 0x4671ecea22c745527033bb9ab9dae0879724013f3e8998b38114522d07f862eb signature: 0x59f6f99fd59c7bbb0aed1301088996953c7ceb67a337ef48f697f0d64aa8f0e42c320dc59fae946dc957be34738404eae272f3539350cea7dbf6e17851d97e6c1b sender: 0xFEbfb5a3756550e6F18Aa4B15370C8642482753D canonical_identifier: CanonicalIdentifier(chain_identifier=5, token_network_address=0xDa1fBc048f503635950058953f5c60FC1F564ee6, channel_identifier=1129) balance_hash: 0x783ad7fec10ccb442d179ddcea9edc5b2f7f435538a24a07a0c51760571c2257 >, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=10), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 11:15:51.037908"}

Node 2

{"message": "PFSFeeUpdate(signature=b\"\\x80\\xc2\\xba\\xae\\xb4\\x8f\\xf8\\x8d\\xbb;W\\x95U\\xc9\\xc8)p\\xf3\\x97+\\xd4E\\xb3r\\x14\\x90\\xa1\\x8c,\\xf6\\x1cO4\\xf2\\xb6\\x8ac\\xc2\\x07\\x04\\xa2\\xce0\\x19 \\x93\\xc3'?\\xe4\\x8aN\\x8a\\t\\xab\\x01\\x12\\xd2\\x8f\\xb4-bD\\x8e\\x1c\", canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), updating_participant=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 38, 6, 683894, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), our_state=NettingChannelEndState(address=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", contract_balance=0, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:38:06.685238"}
{"message": "PFSFeeUpdate(signature=b'o\\x8e^\\xf6\\x1c\\xe8\\xfe\\x0b\\xff\\x01\\xc7\\x98Q\\x96\\xf7\\x01*\\x97\\x153\\xad\\xe4\\x11\\xa5W;\\x13as\\xd2\\xe8a \\xf4R\\xdd\\xed\\xc0\\xbd\\xb4\\xa3\\xc2\\xeb\\x9fs\\xce\\xdc\\xd5\\xa1~\\xe3\\x83x\\xf5\\x15,>f\\x15\\xd6Y\\r/+\\x1c', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1131), updating_participant=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 38, 6, 688340, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1131), our_state=NettingChannelEndState(address=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', contract_balance=0, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:38:06.689505"}
{"message": "PFSFeeUpdate(signature=b\"\\x9b\\xc3\\xb8\\xcc\\xb0&* 4\\x08\\xe4\\xc66\\x90\\x10\\xff\\x0c\\x908'm\\x0b\\xa0cj9\\xe2\\x1aa\\xa3\\xc8\\x8dtf\\x89\\x9e\\x1a\\xb7\\xc5\\xc8\\xc5\\xaa\\x83\\x80%\\xfc5Ts\\x8a\\x1e(\\x98\\xf4\\x87\\x01\\xbe! \\xe1\\xd9\\xb5\\xd2\\x81\\x1c\", canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), updating_participant=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 45, 10, 664291, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), our_state=NettingChannelEndState(address=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:45:10.665281"}
{"message": "PFSFeeUpdate(signature=b\"\\xa2;\\rc's\\xee)\\xc9\\xa5\\xf0\\xb9\\x9c.t\\xa1\\x19Wk\\x1b\\x9a\\x0c\\xdb?\\x95\\x98\\x81\\x18\\xa3\\xa8w\\x82jJ\\xa3 \\x08\\xfe\\xe2\\x90\\xca\\x9b\\xe7Eq\\x91\\xa1d\\xd5Gvv \\xeb\\xb5\\xe6\\x8a\\x1f\\xe8\\xb6\\x10\\xea\\x13\\x97\\x1b\", canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1131), updating_participant=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 45, 10, 667630, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1131), our_state=NettingChannelEndState(address=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'\\xfc\\xdd\\xfcJ\\xc4\\xeaki@\\xf6\\x87\\xb9\\xe7\\xdez&\\xe0\\x8a\\x06\\xcc', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:45:10.668657"}
{"message": "PFSFeeUpdate(signature=b'\\x19\\x8c:\\x99\\xb7\\xae\\xa4x\\xe3\\xfb\\x8fgF2\\xee\\xf6+\\xb6\\xc53\\xcd}\\x02\\x17\\xc6_~9x;\\xb0\\xbc|\\x94\\xab\\xbbB\\x9f\\xfeor0\\xd6\\xaf0\\xcdREO\\xe3\\xfe\\xd2\\xd7\\x0fOi\\xf9o\\xc5#\\xbb\\xd3\\xb8V\\x1b', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), updating_participant=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 11, 1, 11, 744454, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), our_state=NettingChannelEndState(address=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', contract_balance=1000000000000000000, onchain_total_withdraw=100000000000000000, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=1), partner_state=NettingChannelEndState(address=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=1), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 11:01:11.746043"}
{"message": "PFSFeeUpdate(signature=b'\\xb8\\xfe\\xc6L\\x88\\xa7\\xbb\\xaf\\x03(\\xf8J\\xf6L\\x1c\\xae1\\xeb\\xcd\\xecN\\x1dC\\xf0\\x80K\\xf4\\xcc\\xdf\\x0fZ\\xa9M\\x86\\xf6\\xb2\\xbd2l[\\xa1\"K\\xea\\xa1\\xe6P\\xa5\\xc2dYz\\xd2\\xf3\\xaf@1i\\xe0\\x10nn\\x7f \\x1b', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), updating_participant=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 11, 10, 41, 323785, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), our_state=NettingChannelEndState(address=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', contract_balance=1100000000000000000, onchain_total_withdraw=100000000000000000, balance_proof=BalanceProofUnsignedState(nonce=21, transferred_amount=10000000000000000, locked_amount=0, locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), balance_hash=HexBytes('0xdac78515be87e45c9318c6924fb5c03b65eab7bdbd5967344d09e69a97c80d07')), onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=21), partner_state=NettingChannelEndState(address=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=1), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 11:10:41.324811"}

Node 3

{"message": "PFSFeeUpdate(signature=b'6\\x1a&4r2\\xe0\\xaf\\x1a\\xf8\\xd80u\\x82\\xbc\\xefwG\\t\\xc2\\xecKc~\\x1f;l\\xe7 >\\x0b\"L\\xbe#\\xb1[YU\\x04\\x05\\x0ex\\x1c\\xd0P\\xf3bX2\\xb0e%\\xbc\\xf3\\x0f\\xfa\\xe2!\\x8e\\xd4\\x0e\\x90\\x03\\x1c', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1130), updating_participant=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 38, 8, 966455, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1130), our_state=NettingChannelEndState(address=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", contract_balance=0, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'6A\\xcfqg\\xfb?S3\\x9f\\x84\\xa35\\xc7\\xeb\\x7f\\x0e=B\\xf7', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:38:08.967738"}
{"message": "PFSFeeUpdate(signature=b'xf\\xc0\\xec\\xe4i}=\\xdb\\x83\\x95\\x1b\\x1dp\\xf4^\\xf9T\\x1d\\x99\\xe0\\\\\\xeaC\\xad\\x92\\xf0\\x02\\xa3\\x81\\xe7\\xae=m\\x0b\\xc5\\xa5\\xfc4\\x1d\\xddY\\x16w\\xd9\\x00\\x1a\\x11\\x1d5S\\x1b\\xe1\\x1f\\xae\\xfa\\xda\\t[\\xb0\\x8f\\xc1~_\\x1b', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), updating_participant=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 38, 8, 970516, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), our_state=NettingChannelEndState(address=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", contract_balance=0, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:38:08.971606"}
{"message": "PFSFeeUpdate(signature=b'\\xe4K\\xaf_\\xbb\\xb7\\xe3\\ra\\xe7\"\\xde\\xa0\\xe6\\x9401\\x15\\xcc\\xda\\xf4\\xe5\\xcc\\xf8u\\x81\\xad|\\xaae\\xc3\\xedD; \\xd3@zD3\\xacV\\xb0\\xaf\\xd7#o\\x15n\\'0\\x04s\\xc5\\x8b\\x97T\\xa3\\x18\\x0c\\xde@\\xbaY\\x1c', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), updating_participant=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 45, 9, 185877, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), our_state=NettingChannelEndState(address=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:45:09.186949"}
{"message": "PFSFeeUpdate(signature=b\"~\\x06\\tG\\xaeY&\\x96\\x95+Ce;5\\xe1\\xe7-\\x82M'\\xf1Ov\\xe7W\\x88\\xf6yu\\xbb\\xddf\\x7f)\\xbbNI%\\xac\\xad\\x1a\\xd8\\xc7\\xec\\xbb\\xda\\xd5V \\x93*\\xbcg\\xc30\\x05\\x1d\\x1e\\xc5P\\xd6\\xda\\x81\\xaf\\x1b\", canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1130), updating_participant=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 48, 21, 747662, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1130), our_state=NettingChannelEndState(address=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'6A\\xcfqg\\xfb?S3\\x9f\\x84\\xa35\\xc7\\xeb\\x7f\\x0e=B\\xf7', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:48:21.748772"}
{"message": "PFSFeeUpdate(signature=b'\\x8c\\xfcy\\x85\\x93=\\x9b\\xb2\\xc5C]\\xd4!)\\x87E\\xf1\\x97t\\xbf\\xdc\\xa0\\x0c\\x81\\xd4H\\xf6P\\x9c\\x15\\xc3\\xa6NH\\xe2\\x16\\x85\\x15\\x0c\\xd4~q\\x8e/0j\\x8b<\\x17\\xf9\\xfc`Z\\x12u\\x8b\\x15L\\xa6\\xfa^_\\xf5\\x9d\\x1b', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), updating_participant=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 11, 1, 8, 25917, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), our_state=NettingChannelEndState(address=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=1), partner_state=NettingChannelEndState(address=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', contract_balance=1000000000000000000, onchain_total_withdraw=100000000000000000, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=1), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 11:01:08.027060"}
{"message": "PFSFeeUpdate(signature=b'<\\x7fU\\x1b\\xccz\\xce\\xe3\\xb6\\xfe^\\xdc\\xb8\\xd6\\xf7\\xb0\\xc0\\xf0\\xc8\\xd8W(\\xab\\xa9\\x00\\xa5\\x9cWR\\xa9Q\\x19o]\\x14\\x9d2\\xed KX5\\xdf(W\\xa4Bd\\x1fu\\xd64H\\xd4\\x84(\\x1c\\xf9\\x02\\xe8\\xb0\\xa1\\x0e\\xa5\\x1b', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), updating_participant=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 11, 10, 40, 127993, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1128), our_state=NettingChannelEndState(address=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=1), partner_state=NettingChannelEndState(address=b'\\xd8\\xd6\\x91h\\xc7\\xb9C\\xbd\\xefl\\x85x\\x92 u\\x87/\\xde\\x90\\xde', contract_balance=1100000000000000000, onchain_total_withdraw=100000000000000000, balance_proof=BalanceProofSignedState< nonce: 21 transferred_amount: 10000000000000000 locked_amount: 0 locksroot: 0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470 message_hash: 0xf929f01b67bc2910c6bef31d24eb0b11b224efd42b8f07310f70a60114c6f880 signature: 0x413629e2afe9551e173ae52072a433b2d5e88446eee87d8efcc3fe08592b3c23754870b8067fefd3d9c56891c18befe7a8f76103e73bddd9489dd98e31086b261b sender: 0xd8D69168C7b943bdEF6C8578922075872fDE90De canonical_identifier: CanonicalIdentifier(chain_identifier=5, token_network_address=0xDa1fBc048f503635950058953f5c60FC1F564ee6, channel_identifier=1128) balance_hash: 0xdac78515be87e45c9318c6924fb5c03b65eab7bdbd5967344d09e69a97c80d07 >, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=21), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 11:10:40.129380"}

Node 4

{"message": "PFSFeeUpdate(signature=b'\\x8b\\xa3\\x16\\xe0\\x99\\xed\\x8e_\\x19,\\x11\\x02\\xee\\x11\\xe6\\xb4@\\xc3\\x88\\xf0\\xc03\\xb6\\x9e\\xf8\\xa2 O\\xbc%^6\\x1d\\xf0\\xe4\\x19\\xd8|g\\xa9>\\x17?\\xc3\\x86VP\\xc4\\xdf5r3?\\xe0\\x92|M\\xa1l\\xd0dZ\\xda\\x1d\\x1c', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1130), updating_participant=b'6A\\xcfqg\\xfb?S3\\x9f\\x84\\xa35\\xc7\\xeb\\x7f\\x0e=B\\xf7', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 38, 9, 959557, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1130), our_state=NettingChannelEndState(address=b'6A\\xcfqg\\xfb?S3\\x9f\\x84\\xa35\\xc7\\xeb\\x7f\\x0e=B\\xf7', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", contract_balance=0, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:38:09.960845"}
{"message": "PFSFeeUpdate(signature=b'\\xf6\\xd6E$\\xe6\\xf3\\x85\\x99\\xe6\\x9c\\xa6\\x87\\xdd\\xd2\\x08\\xe0,\\x14B\\x13\\xd9\\x1ece\\xba\\xcc\\xe5\\xe9r\\xdc1\\xa5{hw[G\\xea\\\\\\x16m\\xd3\\xab9\\xfc\\x0b\\x05\\xf80\\xa8\\xd8\\x13)\\xc1\\x0b\\xbaq\\x1f\\xfb\\x1b\\x96\\x83\\xbb\\x08\\x1b', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1132), updating_participant=b'6A\\xcfqg\\xfb?S3\\x9f\\x84\\xa35\\xc7\\xeb\\x7f\\x0e=B\\xf7', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 41, 52, 28862, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1132), our_state=NettingChannelEndState(address=b'6A\\xcfqg\\xfb?S3\\x9f\\x84\\xa35\\xc7\\xeb\\x7f\\x0e=B\\xf7', contract_balance=0, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560860, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:41:52.030167"}
{"message": "PFSFeeUpdate(signature=b\"`\\xfa\\xb7LJ\\x91A\\xb8\\xf5j9d\\xf6\\xe3\\xf2\\xe1\\xe6\\xc3\\xa90\\x033)&\\x85\\xec\\x9f\\xb5L1\\x0e\\xefK&\\xd2\\x84n\\xeb\\x90'\\x99\\xd4dr\\x07y%K\\x03\\x7f\\xa1\\xee\\x18\\xcb\\xc7\\xf2\\xd3\\xfa\\xc2;j\\x96Z.\\x1b\", canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1132), updating_participant=b'6A\\xcfqg\\xfb?S3\\x9f\\x84\\xa35\\xc7\\xeb\\x7f\\x0e=B\\xf7', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 45, 10, 8660, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1132), our_state=NettingChannelEndState(address=b'6A\\xcfqg\\xfb?S3\\x9f\\x84\\xa35\\xc7\\xeb\\x7f\\x0e=B\\xf7', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b'\\xfe\\xbf\\xb5\\xa3ueP\\xe6\\xf1\\x8a\\xa4\\xb1Sp\\xc8d$\\x82u=', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560860, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:45:10.010334"}
{"message": "PFSFeeUpdate(signature=b' zq%O\\x8b\\xb4\\x92\\x0c\\xcc\\n\\xb7\\xa8\\x10\\x10\\xa4v\\xdd\\xbe,U\\xa5\\xa5B\\xba\\x12\\t\\xc53\\xde\\xb7\\xd5q\\xb6\\xa5\\xec\\xa3c\\x85t?t\\x8ad\\x0c\\x81\\xb9\\xcb\\xa4\\x1cS!\\x06\\xaa\\xcc4\\xbb\\x81\\x8fT2\\x0f3\\xb7\\x1c', canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1130), updating_participant=b'6A\\xcfqg\\xfb?S3\\x9f\\x84\\xa35\\xc7\\xeb\\x7f\\x0e=B\\xf7', fee_schedule=FeeScheduleState(cap_fees=True, flat=0, proportional=0, imbalance_penalty=None), timestamp=datetime.datetime(2019, 10, 30, 10, 48, 23, 316812, tzinfo=datetime.timezone.utc))", "channel_state": "NettingChannelState(canonical_identifier=CanonicalIdentifier(chain_identifier=5, token_network_address=b'\\xda\\x1f\\xbc\\x04\\x8fP65\\x95\\x00X\\x95?\\\\`\\xfc\\x1fVN\\xe6', channel_identifier=1130), our_state=NettingChannelEndState(address=b'6A\\xcfqg\\xfb?S3\\x9f\\x84\\xa35\\xc7\\xeb\\x7f\\x0e=B\\xf7', contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), partner_state=NettingChannelEndState(address=b\"E\\xe3\\x848\\x91\\xb7-\\xf5\\xda\\n\\xc0\\r\\xc4'1i\\x9a\\x13;\\x86\", contract_balance=1000000000000000000, onchain_total_withdraw=0, balance_proof=None, onchain_locksroot=b\"\\xc5\\xd2F\\x01\\x86\\xf7#<\\x92~}\\xb2\\xdc\\xc7\\x03\\xc0\\xe5\\x00\\xb6S\\xca\\x82';{\\xfa\\xd8\\x04]\\x85\\xa4p\", nonce=0), open_transaction=TransactionExecutionStatus(started_block_number=None, finished_block_number=1560854, result='success'), close_transaction=None, settle_transaction=None, update_transaction=None)", "event": "Sent a PFS Fee Update", "logger": "raiden.services", "level": "debug", "timestamp": "2019-10-30 10:48:23.317890"}

Checking conclusion

By checking the flat, proportional and imbalance_penalty of all the above log events you will see it's always zero fee updates. So I suppose we can rule out a rogue fee update from one of the nodes.

So the question seems to lie with the PFS as theorized in the previous post above.

LefterisJP commented 5 years ago

Reason why node 3 puts a fee on the 99th transfer

Hehe jumped the gun above! The problem does indeed lie with the PFS but it does not ever return non-zero fees. The 99th request to the PFS fails and the node falls back to internal routing which adds a fee estimate because it can't ask the PFS.

Here are the relevant log lines:

{"log_message": "Could not connect to Pathfinding Service (HTTPSConnectionPool(host='pfs-goerli-with-fee.services-dev.raiden.network', port=443): Read timed out. (read timeout=1.0))", "parameters": {"from": "0x45e3843891B72Df5DA0AC00dc42731699a133b86", "to": "0xFEbfb5a3756550e6F18Aa4B15370C8642482753D", "value": 1000000000000000, "max_paths": 5, "iou": {"sender": "0x45e3843891B72Df5DA0AC00dc42731699a133b86", "receiver": "0x062C12c01D0f17fC9eAa33940D994594d91a0182", "one_to_n_address": "0xB3C76048dB9Fe3092a7F88CeEBC580CcbC223A46", "amount": 8800, "expiration_block": 1760881, "chain_id": 5, "signature": "0x3bf7f1b26a4073555d3658c73ac1bd3d8e55cab311675edaf93891af64db7b56528fc6f655a82e337af43e1ce929778f6b0bd40d55790b2630648db99bf3f55c1c"}}, "event": "An error with the path request occured", "logger": "raiden.routing", "level": "warning", "timestamp": "2019-10-30 12:02:01.665700", "exception": "Traceback (most recent call last):\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 387, in _make_request\n    six.raise_from(e, None)\n  File \"<string>\", line 2, in raise_from\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 383, in _make_request\n    httplib_response = conn.getresponse()\n  File \"/usr/lib64/python3.7/http/client.py\", line 1336, in getresponse\n    response.begin()\n  File \"/usr/lib64/python3.7/http/client.py\", line 306, in begin\n    version, status, reason = self._read_status()\n  File \"/usr/lib64/python3.7/http/client.py\", line 267, in _read_status\n    line = str(self.fp.readline(_MAXLINE + 1), \"iso-8859-1\")\n  File \"/usr/lib64/python3.7/socket.py\", line 589, in readinto\n    return self._sock.recv_into(b)\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/gevent/_ssl3.py\", line 503, in recv_into\n    return self.read(nbytes, buffer)\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/gevent/_ssl3.py\", line 332, in read\n    self._wait(self._read_event, timeout_exc=_SSLErrorReadTimeout)\n  File \"src/gevent/_hub_primitives.py\", line 265, in gevent.__hub_primitives.wait_on_socket\n  File \"src/gevent/_hub_primitives.py\", line 266, in gevent.__hub_primitives.wait_on_socket\n  File \"src/gevent/_hub_primitives.py\", line 261, in gevent.__hub_primitives._primitive_wait\n  File \"src/gevent/_hub_primitives.py\", line 262, in gevent.__hub_primitives._primitive_wait\n  File \"src/gevent/_hub_primitives.py\", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  File \"src/gevent/_hub_primitives.py\", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  File \"src/gevent/_hub_primitives.py\", line 55, in gevent.__hub_primitives.WaitOperationsGreenlet.wait\n  File \"src/gevent/_waiter.py\", line 151, in gevent.__waiter.Waiter.get\n  File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/_greenlet_primitives.py\", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch\n  File \"src/gevent/__greenlet_primitives.pxd\", line 35, in gevent.__greenlet_primitives._greenlet_switch\nsocket.timeout: The read operation timed out\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/requests/adapters.py\", line 449, in send\n    timeout=timeout\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 641, in urlopen\n    _stacktrace=sys.exc_info()[2])\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/util/retry.py\", line 368, in increment\n    raise six.reraise(type(error), error, _stacktrace)\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/packages/six.py\", line 686, in reraise\n    raise value\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 603, in urlopen\n    chunked=chunked)\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 389, in _make_request\n    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 307, in _raise_timeout\n    raise ReadTimeoutError(self, url, \"Read timed out. (read timeout=%s)\" % timeout_value)\nurllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='pfs-goerli-with-fee.services-dev.raiden.network', port=443): Read timed out. (read timeout=1.0)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/raiden/network/pathfinding.py\", line 494, in post_pfs_paths\n    timeout=DEFAULT_HTTP_REQUEST_TIMEOUT,\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/requests/api.py\", line 116, in post\n    return request('post', url, data=data, json=json, **kwargs)\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/requests/api.py\", line 60, in request\n    return session.request(method=method, url=url, **kwargs)\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/requests/sessions.py\", line 533, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/requests/sessions.py\", line 646, in send\n    r = adapter.send(request, **kwargs)\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/requests/adapters.py\", line 529, in send\n    raise ReadTimeout(e, request=request)\nrequests.exceptions.ReadTimeout: HTTPSConnectionPool(host='pfs-goerli-with-fee.services-dev.raiden.network', port=443): Read timed out. (read timeout=1.0)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/raiden/routing.py\", line 254, in get_best_routes_pfs\n    value=amount,\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/raiden/network/pathfinding.py\", line 586, in query_paths\n    payload=payload,\n  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/raiden/network/pathfinding.py\", line 499, in post_pfs_paths\n    dict(parameters=payload, exc_info=True),\nraiden.exceptions.ServiceRequestFailed: (\"Could not connect to Pathfinding Service (HTTPSConnectionPool(host='pfs-goerli-with-fee.services-dev.raiden.network', port=443): Read timed out. (read timeout=1.0))\", {'parameters': {'from': '0x45e3843891B72Df5DA0AC00dc42731699a133b86', 'to': '0xFEbfb5a3756550e6F18Aa4B15370C8642482753D', 'value': 1000000000000000, 'max_paths': 5, 'iou': {'sender': '0x45e3843891B72Df5DA0AC00dc42731699a133b86', 'receiver': '0x062C12c01D0f17fC9eAa33940D994594d91a0182', 'one_to_n_address': '0xB3C76048dB9Fe3092a7F88CeEBC580CcbC223A46', 'amount': 8800, 'expiration_block': 1760881, 'chain_id': 5, 'signature': '0x3bf7f1b26a4073555d3658c73ac1bd3d8e55cab311675edaf93891af64db7b56528fc6f655a82e337af43e1ce929778f6b0bd40d55790b2630648db99bf3f55c1c'}}, 'exc_info': True})"}
{"event": "Request to Pathfinding Service was not successful, falling back to internal routing.", "logger": "raiden.routing", "level": "warning", "timestamp": "2019-10-30 12:02:01.674148"}
{"node": "0x45e3843891B72Df5DA0AC00dc42731699a133b86", "state_changes": [{"transfer": {"token_network_registry_address": "0x62436bbadB629498107278486C2A367FfdFEb2f3", "lock_timeout": 30, "secrethash": "0x17847fe436f0ab704be4df73621a90aaaa194b2df61eef4a9bfd29326d7db98a", "token_network_address": "0xDa1fBc048f503635950058953f5c60FC1F564ee6", "payment_identifier": "15188145939394927688", "amount": "1000000000000000", "secret": "<redacted>", "initiator": "0x45e3843891B72Df5DA0AC00dc42731699a133b86", "target": "0xFEbfb5a3756550e6F18Aa4B15370C8642482753D"}, "routes": [{"forward_channel_id": "1128", "route": ["0x45e3843891B72Df5DA0AC00dc42731699a133b86", "0xd8D69168C7b943bdEF6C8578922075872fDE90De", "0xFCDdFC4AC4ea6b6940f687B9e7De7a26e08a06CC", "0xFEbfb5a3756550e6F18Aa4B15370C8642482753D"], "estimated_fee": "20000000000000"}, {"forward_channel_id": "1130", "route": ["0x45e3843891B72Df5DA0AC00dc42731699a133b86", "0x3641Cf7167fB3f53339f84A335C7EB7f0E3d42f7", "0x45e3843891B72Df5DA0AC00dc42731699a133b86", "0xd8D69168C7b943bdEF6C8578922075872fDE90De", "0xFCDdFC4AC4ea6b6940f687B9e7De7a26e08a06CC", "0xFEbfb5a3756550e6F18Aa4B15370C8642482753D"], "estimated_fee": "20000000000000"}], "_type": "raiden.transfer.mediated_transfer.state_change.ActionInitInitiator"}], "event": "State changes", "logger": "raiden.raiden_service", "level": "debug", "timestamp": "2019-10-30 12:02:01.676104"}

The exception raised (posted below in a more readable format) shows that the PFS read times out:

Traceback (most recent call last):
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 387, in _make_request
    six.raise_from(e, None)
  File \"<string>\", line 2, in raise_from
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 383, in _make_request
    httplib_response = conn.getresponse()
  File \"/usr/lib64/python3.7/http/client.py\", line 1336, in getresponse
    response.begin()
  File \"/usr/lib64/python3.7/http/client.py\", line 306, in begin
    version, status, reason = self._read_status()
  File \"/usr/lib64/python3.7/http/client.py\", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), \"iso-8859-1\")
  File \"/usr/lib64/python3.7/socket.py\", line 589, in readinto
    return self._sock.recv_into(b)
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/gevent/_ssl3.py\", line 503, in recv_into
    return self.read(nbytes, buffer)
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/gevent/_ssl3.py\", line 332, in read
    self._wait(self._read_event, timeout_exc=_SSLErrorReadTimeout)
  File \"src/gevent/_hub_primitives.py\", line 265, in gevent.__hub_primitives.wait_on_socket
  File \"src/gevent/_hub_primitives.py\", line 266, in gevent.__hub_primitives.wait_on_socket
  File \"src/gevent/_hub_primitives.py\", line 261, in gevent.__hub_primitives._primitive_wait
  File \"src/gevent/_hub_primitives.py\", line 262, in gevent.__hub_primitives._primitive_wait
  File \"src/gevent/_hub_primitives.py\", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File \"src/gevent/_hub_primitives.py\", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File \"src/gevent/_hub_primitives.py\", line 55, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File \"src/gevent/_waiter.py\", line 151, in gevent.__waiter.Waiter.get
  File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File \"src/gevent/_greenlet_primitives.py\", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File \"src/gevent/_greenlet_primitives.py\", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File \"src/gevent/__greenlet_primitives.pxd\", line 35, in gevent.__greenlet_primitives._greenlet_switch
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/requests/adapters.py\", line 449, in send
    timeout=timeout
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/util/retry.py\", line 368, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/packages/six.py\", line 686, in reraise
    raise value
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 603, in urlopen
    chunked=chunked)
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 389, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/urllib3/connectionpool.py\", line 307, in _raise_timeout
    raise ReadTimeoutError(self, url, \"Read timed out. (read timeout=%s)\" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='pfs-goerli-with-fee.services-dev.raiden.network', port=443): Read timed out. (read timeout=1.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/raiden/network/pathfinding.py\", line 494, in post_pfs_paths
    timeout=DEFAULT_HTTP_REQUEST_TIMEOUT,
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/requests/api.py\", line 116, in post
    return request('post', url, data=data, json=json, **kwargs)
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/requests/api.py\", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/requests/sessions.py\", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/requests/sessions.py\", line 646, in send
    r = adapter.send(request, **kwargs)
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/requests/adapters.py\", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='pfs-goerli-with-fee.services-dev.raiden.network', port=443): Read timed out. (read timeout=1.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/raiden/routing.py\", line 254, in get_best_routes_pfs
    value=amount,
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/raiden/network/pathfinding.py\", line 586, in query_paths
    payload=payload,
  File \"/home/lefteris/.virtualenvs/scenarioplayer/lib/python3.7/site-packages/raiden/network/pathfinding.py\", line 499, in post_pfs_paths
    dict(parameters=payload, exc_info=True),
raiden.exceptions.ServiceRequestFailed: (\"Could not connect to Pathfinding Service (HTTPSConnectionPool(host='pfs-goerli-with-fee.services-dev.raiden.network', port=443): Read timed out. (read timeout=1.0))\", {'parameters': {'from': '0x45e3843891B72Df5DA0AC00dc42731699a133b86', 'to': '0xFEbfb5a3756550e6F18Aa4B15370C8642482753D', 'value': 1000000000000000, 'max_paths': 5, 'iou': {'sender': '0x45e3843891B72Df5DA0AC00dc42731699a133b86', 'receiver': '0x062C12c01D0f17fC9eAa33940D994594d91a0182', 'one_to_n_address': '0xB3C76048dB9Fe3092a7F88CeEBC580CcbC223A46', 'amount': 8800, 'expiration_block': 1760881, 'chain_id': 5, 'signature': '0x3bf7f1b26a4073555d3658c73ac1bd3d8e55cab311675edaf93891af64db7b56528fc6f655a82e337af43e1ce929778f6b0bd40d55790b2630648db99bf3f55c1c'}}

Next question

Why does the 99th request to the PFS in this scenario time out? I may need help from @karlb here.

Dominik1999 commented 5 years ago

I like your bug descriptions @LefterisJP it is a good read

Dominik1999 commented 5 years ago

LefterisJP commented 5 years ago

Why does the 99th request to the PFS in this scenario time out? I may need help from @karlb here.

We had a look at the PFS logs with @karlb.

Turns out that for this particular request there is not even a log entry at the PFS side. That means that the request never made it to the PFS but timed out instead.

The theory for that is that since many requests are sent to the PFS at the same time and the pathfinding timeout is only 1 second

https://github.com/raiden-network/raiden/blob/1ff6e6e22017b16a22fcef0e23535d5c41c617cc/raiden/constants.py#L153

then one of the requests does not make it within this one second window and is killed by the node before even reaching the PFS. Causing it to revert back to internal routing as explained here.

Increasing the request timeout fixes this for a local scenario run.

Finally to make the scenario pass we wait for 40 blocks to make sure that the settlement timeout has passed and the channel0-4 is settled and as such taken out of the balance calculation. And we adjust the asserted balance to 1_105_000_000_000_000_000.

raiden-network / raiden