search

raiden-network / raiden

Raiden Network
https://developer.raiden.network
Other
1.84k stars 377 forks source link

Crash during startup on mainnet #3558

Closed palango closed 5 years ago

palango commented 5 years ago

Problem Definition

I was able to reproduce a problem forwarded to us. I only see this problem on mainnet, but I'm currently not sure if it's related to the production flag (that one works fine on rinkeby testnet) or somehow related to the matrix server (@ulope guess) or that we try to sync to big intervals (I had the problem of web3 requests timing out for the MS).

./raiden-v0.100.2-macOS-x86_64 --network-id mainnet --eth-rpc-endpoint http://geth.mainnet.ethnodes.brainbot.com:8545 --gas-price fast --accept-disclaimer
Welcome to Raiden, version 0.100.2!
----------------------------------------------------------------------
| This is an Alpha version of experimental open source software      |
| released as a test version under an MIT license and may contain    |
| errors and/or bugs. No guarantee or representations whatsoever is  |
| made regarding its suitability (or its use) for any purpose or     |
| regarding its compliance with any applicable laws and regulations. |
| Use of the software is at your own risk and discretion and by      |
| using the software you acknowledge that you have read this         |
| disclaimer, understand its contents, assume all risk related       |
| thereto and hereby release, waive, discharge and covenant not to   |
| sue Brainbot Labs Establishment or any officers, employees or      |
| affiliates from and for any direct or indirect liability resulting |
| from the use of the software as permissible by applicable laws and |
| regulations.                                                       |
|                                                                    |
| Privacy Warning: Please be aware, that by using the Raiden Client, |
| among others, your Ethereum address, channels, channel deposits,   |
| settlements and the Ethereum address of your channel counterparty  |
| will be stored on the Ethereum chain, i.e. on servers of Ethereum  |
| node operators and ergo are to a certain extent publicly available.|
| The same might also be stored on systems of parties running Raiden |
| nodes connected to the same token network. Data present in the     |
| Ethereum chain is very unlikely to be able to be changed, removed  |
| or deleted from the public arena.                                  |
|                                                                    |
| Also be aware, that data on individual Raiden token transfers will |
| be made available via the Matrix protocol to the recipient,        |
| intermediating nodes of a specific transfer as well as to the      |
| Matrix server operators.                                           |
----------------------------------------------------------------------
2019-02-27 11:05:29.879284 [info     ] Starting Raiden                [raiden.ui.runners] distribution=bundled python_implementation=CPython python_version=3.7.0 raiden=0.100.2 system=macOS 10.14.3 64bit

Select one of them by index to continue: 0
Enter the password to unlock 0xXXXXXXXXXXX:
Raiden is running in production mode
Checking if the ethereum node is synchronized

You are connected to the 'mainnet' network and the DB path is: /Users/paul/.raiden/node_28898a6b/netid_1/network_b9633dd9/v17_log.db
2019-02-27 11:05:42.700442 [info     ] Automatically selecting matrix homeserver based on RTT [raiden.network.transport.matrix] homeserver=https://persephone.raidentransport.digitalvirtues.com rtt=0.025393333333333334
Traceback (most recent call last):
  File "urllib3/connectionpool.py", line 384, in _make_request
  File "<string>", line 2, in raise_from
  File "urllib3/connectionpool.py", line 380, in _make_request
  File "http/client.py", line 1321, in getresponse
  File "http/client.py", line 296, in begin
  File "http/client.py", line 257, in _read_status
  File "socket.py", line 589, in readinto
  File "gevent/_socket3.py", line 433, in recv_into
  File "src/gevent/_hub_primitives.py", line 265, in gevent.__hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 266, in gevent.__hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 261, in gevent.__hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 262, in gevent.__hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent.__hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 151, in gevent.__waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 59, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 63, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "requests/adapters.py", line 449, in send
  File "urllib3/connectionpool.py", line 638, in urlopen
  File "urllib3/util/retry.py", line 367, in increment
  File "urllib3/packages/six.py", line 686, in reraise
  File "urllib3/connectionpool.py", line 600, in urlopen
  File "urllib3/connectionpool.py", line 386, in _make_request
  File "urllib3/connectionpool.py", line 306, in _raise_timeout
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='geth.mainnet.ethnodes.brainbot.com', port=8545): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "build/raiden/raiden-script.py", line 2, in <module>
  File "raiden/__main__.py", line 11, in main
  File "click/core.py", line 722, in __call__
  File "click/core.py", line 697, in main
  File "click/core.py", line 1043, in invoke
  File "click/core.py", line 895, in invoke
  File "click/core.py", line 535, in invoke
  File "click/decorators.py", line 17, in new_func
  File "raiden/ui/cli.py", line 439, in run
  File "raiden/ui/runners.py", line 252, in run
  File "raiden/ui/runners.py", line 96, in _start_services
  File "raiden/ui/app.py", line 418, in run_app
  File "raiden/app.py", line 132, in start
  File "raiden/raiden_service.py", line 370, in start
  File "raiden/tasks.py", line 182, in first_run
  File "raiden/tasks.py", line 222, in _maybe_run_callbacks
  File "raiden/raiden_service.py", line 555, in _callback_new_block
  File "raiden/blockchain/events.py", line 217, in poll_blockchain_events
  File "raiden/utils/filters.py", line 117, in get_new_entries
  File "web3/eth.py", line 354, in getLogs
  File "web3/manager.py", line 109, in request_blocking
  File "web3/manager.py", line 92, in _make_request
  File "web3/middleware/cache.py", line 105, in middleware
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 48, in apply_formatters
  File "web3/middleware/gas_price_strategy.py", line 18, in middleware
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 48, in apply_formatters
  File "web3/middleware/attrdict.py", line 18, in middleware
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 48, in apply_formatters
  File "web3/middleware/normalize_errors.py", line 9, in middleware
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 50, in apply_formatters
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 48, in apply_formatters
  File "cytoolz/functoolz.pyx", line 232, in cytoolz.functoolz.curry.__call__
  File "web3/middleware/formatting.py", line 50, in apply_formatters
  File "web3/middleware/exception_retry_request.py", line 80, in middleware
  File "web3/providers/rpc.py", line 68, in make_request
  File "web3/utils/request.py", line 26, in make_post_request
  File "requests/sessions.py", line 572, in post
  File "requests/sessions.py", line 524, in request
  File "requests/sessions.py", line 637, in send
  File "requests/adapters.py", line 529, in send
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='geth.mainnet.ethnodes.brainbot.com', port=8545): Read timed out. (read timeout=10)
[88549] Failed to execute script raiden-script
--- Logging error ---
Exception ignored in: <function BaseFileLock.__del__ at 0x10ae878c8>
Traceback (most recent call last):
  File "filelock.py", line 333, in __del__
  File "filelock.py", line 320, in release
  File "logging/__init__.py", line 1332, in info
  File "logging/__init__.py", line 1468, in _log
  File "logging/__init__.py", line 1478, in handle
  File "logging/__init__.py", line 1540, in callHandlers
  File "logging/__init__.py", line 854, in handle
  File "logging/handlers.py", line 72, in emit
  File "logging/__init__.py", line 907, in handleError
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/traceback.py", line 100, in print_exception
  File "raiden/log_config.py", line 138, in tracebackexception_format
  File "raiden/log_config.py", line 123, in processor_wrapper
ImportError: sys.meta_path is None, Python is likely shutting down

System Description

Darwin Pauls-MacBook-Pro-2.local 18.2.0 Darwin Kernel Version 18.2.0: Thu Dec 20 20:46:53 PST 2018; root:xnu-4903.241.1~1/RELEASE_X86_64 x86_64
/usr/local/bin/solc
solc, the solidity compiler commandline interface
Version: 0.5.2+commit.1df8f40c.Darwin.appleclang
/usr/local/bin/geth
Geth
Version: 1.8.22-stable
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.11.5
Operating System: darwin
GOPATH=
GOROOT=/usr/local/Cellar/go/1.11.5/libexec
ulope commented 5 years ago

@palango Actually you're correct, it's not matrix related. I misread the traceback that was posted in Gitter.

The most likely cause of a read timeout is that the node takes too long to generate a response to whatever was requested (likely a filter in this case).

palango commented 5 years ago

The most likely cause of a read timeout is that the node takes too long to generate a response to whatever was requested (likely a filter in this case).

And the cause of this is probably that the whole blockrange from last_known_block to 'latest' is queried at once, whgich can get quite big on the first node start and leads to the timeout of the connection.

palango commented 5 years ago

This happened again to some people on the EthCC booth. Can someone on linux try to reproduce this?

LefterisJP commented 5 years ago

I will pick this up and fix asap.