search

steemit / steem-python

The official Python (3) library for the Steem Blockchain.
https://steem.io
MIT License
154 stars 100 forks source link

steempy backup nodes doesn't work (and you will never know until you need it) #196

Closed Gandalf-the-Grey closed 6 years ago

Gandalf-the-Grey commented 6 years ago

setting up custom node works fine: steempy set nodes https://gtg.steem.house:8090/ setting up such node as backup/fallback node works fine: set nodes https://api.steemit.com,https://gtg.steem.house:8090/ but it does not if it can't connect to first node from the list set nodes https://localhost:8080,https://api.steemit.com,https://gtg.steem.house:8090/ where port 8080 at localhost is closed it won't even try next nodes

gandalf@sandbox:~$ steempy info
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 138, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 98, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 88, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 350, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 837, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 281, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 147, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f6063ae6828>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steembase/http_client.py", line 202, in call
  File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 244, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 671, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 643, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 363, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f6063ae6828>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/steempy", line 11, in <module>
    load_entry_point('steem==1.0.0', 'console_scripts', 'steempy')()
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/cli.py", line 776, in legacyentry
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/steem.py", line 60, in __init__
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/commit.py", line 96, in __init__
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/wallet.py", line 61, in __init__
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/steemd.py", line 65, in chain_params
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/steemd.py", line 419, in get_dynamic_global_properties
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steembase/http_client.py", line 223, in call
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steembase/http_client.py", line 243, in call
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steembase/http_client.py", line 262, in _return
steembase.exceptions.RPCError: Error Response: {'message': 'Assert Exception:api_itr != _registered_apis.end(): Could not find API ', 'data': {'message': 'Assert Exception', 'stack': [{'data': {'api': ''}, 'format': 'api_itr != _registered_apis.end(): Could not find API ${api}', 'context': {'method': 'find_api_method', 'file': 'json_rpc_plugin.cpp', 'level': 'error', 'line': 203, 'hostname': '', 'timestamp': '2018-03-27T21:20:41'}}], 'code': 10, 'name': 'assert_exception'}, 'code': -32002}

bonus: I can't revert from that situation by setting up working node: steempy set nodes https://gtg.steem.house:8090/ for the same reason... steempy fails to set it because it crashes while connecting to current (bad) node (workaround: editing or just removing .local/share/steem/steem.sqlite)

Gandalf-the-Grey commented 6 years ago

Too narrow errorList? https://github.com/steemit/steem-python/blob/9628744f838cdf85ee797db194e6fcc51921c310/steembase/http_client.py#L194-L199

roadscape commented 6 years ago

node handling in this lib needs a lot of work. very unpredictable.

roadscape commented 6 years ago

@Gandalf-the-Grey are you able to reproduce this on master?

cyon1c commented 6 years ago

I can't repro either. Appears to be working as expected. Going to close unless @Gandalf-the-Grey you still see this happening.

roadscape commented 6 years ago

I have not been able to reproduce this issue. Observations:

  1. Why does the cli make a steem request before it will allow you to apply any setting updates??
  2. If I'm reading the recursive stack trace correctly, ConnectionRefusedError was handled fine, and the final error was from steemd: Could not find API -- a valid response which should not be retried.

At this point it would be great if you could try to reproduce the issue. And at any rate we need to fix the CLI behavior but I've been able to put in junk nodes and edit them just fine right now (aside from annoying 60s timeout before new settings are applied)

jnordberg commented 6 years ago

Not seeing any PR for this to review

roadscape commented 6 years ago

@jnordberg no PR yet (may not be necessary), we’ve asked OP to reproduce the behavior since much of this code has been updated lately

Gandalf-the-Grey commented 6 years ago

Unfortunately I can still run into such troubles. However I guess that now I need to search for more tricky cases. All depends on exact issue with connection, sometimes is easily handled like trivial connection refused (tcp reject):

$ steempy set nodes https://127.0.0.1,https://api.steemit.com,https://gtg.steem.house:8090/
$ steempy info
WARNING:urllib3.connectionpool:Retrying (Retry(total=19, connect=None, read=None, redirect=0)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fd62a10b908>: Failed to establish a new connection: [Errno 111] Connection refused',)': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=18, connect=None, read=None, redirect=0)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fd62a10b7f0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /
WARNING:urllib3.connectionpool:Retrying (Retry(total=17, connect=None, read=None, redirect=0)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fd62a10b668>: Failed to establish a new connection: [Errno 111] Connection refused',)': /

(...) up to the limit of retries or even in case of a node that returns some unexpected "garbage"

$ steempy set nodes https://steemit.com,https://api.steemit.com,https://gtg.steem.house:8090/
$ steempy info
Unexpected error: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/local/bin/steempy", line 11, in <module>
    load_entry_point('steem==1.0.0', 'console_scripts', 'steempy')()
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/cli.py", line 776, in legacyentry
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/steem.py", line 60, in __init__
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/commit.py", line 96, in __init__
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/wallet.py", line 61, in __init__
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/steemd.py", line 65, in chain_params
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/steemd.py", line 419, in get_dynamic_global_properties
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steembase/http_client.py", line 271, in call
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steembase/http_client.py", line 231, in call
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

but that can be more tricky with variety of different things that can go wrong while connecting, and once I find it I'm locked out (until I remove that in directly within sqlite) Example:

$ steempy set nodes https://8.8.8.8,https://api.steemit.com,https://gtg.steem.house:8090/
$ steempy info
Unexpected error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 350, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 837, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 323, in connect
    ssl_context=context)
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 324, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.5/ssl.py", line 385, in wrap_socket
    _context=self)
  File "/usr/lib/python3.5/ssl.py", line 760, in __init__
    self.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/steempy", line 11, in <module>
    load_entry_point('steem==1.0.0', 'console_scripts', 'steempy')()
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/cli.py", line 776, in legacyentry
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/steem.py", line 59, in __init__
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/commit.py", line 96, in __init__
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/wallet.py", line 61, in __init__
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/steemd.py", line 65, in chain_params
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steem/steemd.py", line 419, in get_dynamic_global_properties
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steembase/http_client.py", line 291, in call
  File "/usr/local/lib/python3.5/dist-packages/steem-1.0.0-py3.5.egg/steembase/http_client.py", line 235, in call
  File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 244, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 624, in urlopen
    raise SSLError(e)
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720)

And now I'm unable to go back to safe set: steempy set nodes https://api.steemit.com,https://gtg.steem.house:8090/ because it fails at attempts to connect 8.8.8.8 (sorry, google still doesn't run their own Steem API)

Different commands that supposed to be local has troubles too, like steempy config

cyon1c commented 6 years ago

@Gandalf-the-Grey Are you using the most recent version from master? Or the version from PyPi? Using master, the line numbers for your tracebacks don't match up to my codebase.

That being said, I did manage to repro the JSONDecodeError, which is up in PR #225.

We're gonna get this merged and if you can test again using master, let us know if you are still having issues. @roadscape and I suspect that we need to refactor the error handling for retrying errors in http_client further, but this should get us in that direction.

roadscape commented 6 years ago

Between https://github.com/steemit/steem-python/pull/225 and https://github.com/steemit/steem-python/pull/229 we can call this resolved. Should it happen again we'll get more useful information.