Open konradkonrad opened 3 years ago
Can you try manually doing a curl request to both of the PFS' info endpoints from the SP server (ideally from within a docker container)?
I had some slowness when talking to boris' PFS and my current theory is that it's a problem with non-working IPv6.
We can see, that there are about 30 seconds from https://github.com/raiden-network/raiden/blob/develop/raiden/ui/app.py#L383 (which ultimately logs Using Pathfinding Service
) until the request fails. Within these 30 seconds now further log output is generated.
However, we still see some output in the nodes stdout
:
cat /home/konrad/.raiden/scenario-player/scenarios/bf1_basic_functionality/node_0023_000/run-023.stdout
--------- Starting ---------
Command line: /home/konrad/.local/bin/raiden-v2.0.0rc3-linux-x86_64 --accept-disclaimer --datadir /home/konrad/.raiden/scenario-player/scenarios/bf1_basic_functionality/node_0023_000 --keystore-path /home/konrad/.raiden/scenario-player/scenarios/bf1_basic_functionality/node_0023_000/keys --address 0xA1362e9521E08Ba39cb78b8DCa11Af1C82CD669E --password-file /home/konrad/.raiden/scenario-player/scenarios/bf1_basic_functionality/node_0023_000/password.txt --network-id 1 --sync-check --gas-price fast --eth-rpc-endpoint http://10.104.6.23:8545 --log-config :info,raiden:debug,raiden_contracts:debug,raiden.api.rest.pywsgi:warning --log-json --log-file /home/konrad/.raiden/scenario-player/scenarios/bf1_basic_functionality/node_0023_000/run-023.log --disable-debug-logfile --matrix-server https://transport.raiden.badgateway.tech --api-address 127.0.0.1:41265 --no-web-ui --pathfinding-service-address https://pfs.raiden.overdoze.se --flat-fee 0x6B175474E89094C44Da98b954EedeAC495271d0F 0 --proportional-fee 0x6B175474E89094C44Da98b954EedeAC495271d0F 0 --enable-monitoring --gas-price fast --proportional-imbalance-fee 0x6B175474E89094C44Da98b954EedeAC495271d0F 0 --pathfinding-max-fee 50000000000000000 --routing-mode pfs --environment-type production --default-settle-timeout 100 --default-reveal-timeout 50 --pathfinding-max-paths 5
Welcome to Raiden, version 2.0.0rc3!
[...DISCLAIMER...]
The Raiden API RPC server is now running at http://127.0.0.1:41265/.
See the Raiden documentation for all available endpoints at
https://raiden-network.readthedocs.io/en/latest/rest_api.html
Checking if the ethereum node is synchronized
You have chosen the Pathfinding Service at https://pfs.raiden.overdoze.se.
Operator: hymner, running version: 0.15.4, chain_id: 1.
Fees will be paid to 0x025dCC9F2964C1D7F886f7cB55061115e334bF56. Each request costs 0.045 RDN.
Message from the Pathfinding Service:
Overdoze PFS at your service
Raiden is running in production mode
You are connected to the 'mainnet' network and the DB path is: /home/konrad/.raiden/scenario-player/scenarios/bf1_basic_functionality/node_0023_000/node_a1362e95/netid_1/network_6d07a2ef/v27_log.db
This is the first time Raiden is being used with this address. Processing all the events may take some time. Please wait ...
Synchronization complete, REST API services now available.
--------- Starting ---------
Command line: /home/konrad/.local/bin/raiden-v2.0.0rc3-linux-x86_64 --accept-disclaimer --datadir /home/konrad/.raiden/scenario-player/scenarios/bf1_basic_functionality/node_0023_000 --keystore-path /home/konrad/.raiden/scenario-player/scenarios/bf1_basic_functionality/node_0023_000/keys --address 0xA1362e9521E08Ba39cb78b8DCa11Af1C82CD669E --password-file /home/konrad/.raiden/scenario-player/scenarios/bf1_basic_functionality/node_0023_000/password.txt --network-id 1 --sync-check --gas-price fast --eth-rpc-endpoint http://10.104.6.23:8545 --log-config :info,raiden:debug,raiden_contracts:debug,raiden.api.rest.pywsgi:warning --log-json --log-file /home/konrad/.raiden/scenario-player/scenarios/bf1_basic_functionality/node_0023_000/run-023.log --disable-debug-logfile --matrix-server https://transport.raiden.badgateway.tech --api-address 127.0.0.1:41265 --no-web-ui --pathfinding-service-address https://pfs.raiden.overdoze.se --flat-fee 0x6B175474E89094C44Da98b954EedeAC495271d0F 0 --proportional-fee 0x6B175474E89094C44Da98b954EedeAC495271d0F 0 --enable-monitoring --gas-price fast --proportional-imbalance-fee 0x6B175474E89094C44Da98b954EedeAC495271d0F 0 --pathfinding-max-fee 50000000000000000 --routing-mode pfs --environment-type production --default-settle-timeout 100 --default-reveal-timeout 50 --pathfinding-max-paths 5
Welcome to Raiden, version 2.0.0rc3!
[...DISCLAIMER...]
The Raiden API RPC server is now running at http://127.0.0.1:41265/.
See the Raiden documentation for all available endpoints at
https://raiden-network.readthedocs.io/en/latest/rest_api.html
Checking if the ethereum node is synchronized
You have chosen the Pathfinding Service at https://pfs.raiden.overdoze.se.
Operator: hymner, running version: 0.15.4, chain_id: 1.
Fees will be paid to 0x025dCC9F2964C1D7F886f7cB55061115e334bF56. Each request costs 0.045 RDN.
Message from the Pathfinding Service:
Overdoze PFS at your service
Raiden is running in production mode
You are connected to the 'mainnet' network and the DB path is: /home/konrad/.raiden/scenario-player/scenarios/bf1_basic_functionality/node_0023_000/node_a1362e95/netid_1/network_6d07a2ef/v27_log.db
So we know that https://github.com/raiden-network/raiden/blob/develop/raiden/ui/app.py#L410 was still reached. However we did not make it to https://github.com/raiden-network/raiden/blob/develop/raiden/ui/app.py#L423 before the 30 seconds are over.
Can you try manually doing a curl request to both of the PFS' info endpoints from the SP server (ideally from within a docker container)?
I had some slowness when talking to boris' PFS and my current theory is that it's a problem with non-working IPv6.
curl -w "@curl-format.txt" -o /dev/null -s "https://pfs.raiden.overdoze.se/api/v1/info"
time_namelookup: 0.001435s
time_connect: 0.039997s
time_appconnect: 0.105902s
time_pretransfer: 0.105971s
time_redirect: 0.000000s
time_starttransfer: 0.148042s
----------
time_total: 0.148160s
curl -w "@curl-format.txt" -o /dev/null -s "https://pfs.raiden.badgateway.tech/api/v1/info"
time_namelookup: 0.082534s
time_connect: 0.308856s
time_appconnect: 0.359889s
time_pretransfer: 0.359956s
time_redirect: 0.000000s
time_starttransfer: 0.388109s
----------
time_total: 0.388382s
(Using this method)
Ah we crossed messages. Looks like it hangs after the PFS info request. So then that's likely not the problem.
I'm reading very slow responses when connecting through httpie
(i.e. with a python backend) to badgateway.tech
:
# date && http https://transport.raiden.overdoze.se/health && date
Tue Jun 8 12:39:32 PM CEST 2021
HTTP/1.1 200 OK
Content-Length: 2
Content-Type: text/plain
Date: Tue, 08 Jun 2021 10:39:41 GMT
Server: Synapse/1.33.0
OK
Tue Jun 8 12:39:33 PM CEST 2021
# <= 1 second
# date && http https://transport.raiden.badgateway.tech/health && date
Tue Jun 8 12:39:49 PM CEST 2021
HTTP/1.1 200 OK
Content-Length: 2
Content-Type: text/plain
Date: Tue, 08 Jun 2021 10:42:07 GMT
Server: Synapse/1.33.0
OK
Tue Jun 8 12:41:59 PM CEST 2021
# >= 130 seconds!!!
With curl
that second request is instant
During scenario run bf1 the node restart is sometimes slow, too slow for the scenario to succeed:
This leads to an error when requesting the task after restart: