Intermittent `socket.timeout`

[adamkrellenstein]

We're running into a lot of the following errors with our use of python-bitcoinlib in counterpartyd:

Traceback (most recent call last):
  File "C:\counterpartyd_build\dist\counterpartyd\counterpartyd.py", line 956, in <module>
    blocks.follow(db)
  File "C:\counterpartyd_build\dist\counterpartyd\lib\blocks.py", line 977, in follow
    mempool_tx_index = list_tx(db, None, block_index, curr_time, tx_hash, mempool_tx_index)
  File "C:\counterpartyd_build\dist\counterpartyd\lib\blocks.py", line 653, in list_tx
    source, destination, btc_amount, fee, data = get_tx_info(tx_dict['hex'], block_index)
  File "C:\counterpartyd_build\dist\counterpartyd\lib\blocks.py", line 341, in get_tx_info
    tx_info = get_tx_info2(tx_hex, block_parser=block_parser)
  File "C:\counterpartyd_build\dist\counterpartyd\lib\blocks.py", line 560, in get_tx_info2
    vin_ctx = proxy.getrawtransaction(vin.prevout.hash)
  File "C:\counterpartyd_build\env\lib\site-packages\bitcoin\rpc.py", line 307, in getrawtransaction
    r = self._call('getrawtransaction', b2lx(txid), 1 if verbose else 0)
  File "C:\counterpartyd_build\env\lib\site-packages\bitcoin\rpc.py", line 144, in _call
    response = self._get_response()
  File "C:\counterpartyd_build\env\lib\site-packages\bitcoin\rpc.py", line 179, in _get_response
    http_response = self.__conn.getresponse()
  File "C:\Python34\Lib\http\client.py", line 1172, in getresponse
    response.begin()
  File "C:\Python34\Lib\http\client.py", line 351, in begin
    version, status, reason = self._read_status()
  File "C:\Python34\Lib\http\client.py", line 313, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "C:\Python34\Lib\socket.py", line 371, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

Sometimes the problem was being caused by reusing a Proxy object, however, AFAIK, we've gotten rid of all of those cases and the error still shows up infrequently and intermittently. Any idea what could be causing this? Is it really the backend taking too long? (I've tried using both Bitcoin Core and btcd.) Should there just be a retry loop somewhere where there isn't? I've also tried socket.setdefaulttimeout(10). Should we be using python-bitcoinlib otherwise? (Should I just be talking to the socket or httplib teams?

[adamkrellenstein]

Also, setting the timeout variable when initialising Proxy doesn't seem to help either? That's for the connection, not the socket?

[adamkrellenstein]

Nevermind. This was fixed in 0.3.0: 82d6d9579b6dc806e3052f3ac6954ab134849e62

[adamkrellenstein]

Actually, this issue is persisting.

[petertodd]

Curious. Have you tried making a minimal test program that reproduces the behavior? I've never seen that myself, but I rarely use Proxy() for long-standing things.

[ouziel-slama]

@petertodd, seems that a big part of the problem comes from the fact that we are opening too much connections and bitcoind crash. Because python-bitcoinlib is not thread safe, we can not avoid this without adding a parameter proxy to a lot of functions. Are you ok to use the library requests instead httplib here https://github.com/petertodd/python-bitcoinlib/blob/master/bitcoin/rpc.py ? requests is thread safe and supports Python from 2.6 to 3.4.

[petertodd]

@ouziel-slama Ah, so your problem is that your threaded program has sufficient numbers of threads - each with a Proxy object - that you're crashing bitcoind?

Why don't you create a Proxy wrapper that itself sends requests to a single/threadpooled object instead? It'd save a dependency that would only be used by a small niche of users.

What exactly are the majority of these requests? Static blockchain data right?

[ouziel-slama]

@ouziel-slama Ah, so your problem is that your threaded program has sufficient numbers of threads - each with a Proxy object - that you're crashing bitcoind?

with have only 3 threads (the main thread and two another for the API), but we are opening a new connection for each request, if not we are getting errors like http.client.CannotSendRequest: Request-sent

Why don't you create a Proxy wrapper that itself sends requests to a single/threadpooled object instead?

because we have the simpler solution to add a parameter to a lot of functions ;-)

It'd save a dependency that would only be used by a small niche of users.

hum.. I don't think that to be thread safe is a need only for a niche of users. And certainly not for a library that need to make http requests. IMHO this little dependency is widely justified.

[adamkrellenstein]

@petertodd, Yeah, I agree with Ouziel. We can create a Proxy object for each of our three threads and just pass them around, but it seems to me that python-bitcoinlib should just be thread-safe, and that if all you need to achieve that is a dependency on requests, it's worth having it.

(Yes, the requests are of static blockchain data.)

[adamkrellenstein]

Update: we're now doing that---one connection per thread, etc.

[petertodd]

From the feedback I get the vast majority of actual python-bitcoinlib usage is small experiments, quick demo's of features, etc. I'm very hesitant to add dependencies at the risk of making the process of using python-bitcoinlib harder for those people, even if it makes using it for bigger projects less clean in some cases.

BTW there's a way to increase the number of RPC threads:

  -rpcthreads=<n>        Set the number of threads to service RPC calls (default: 4)

Also since the data you need is static blockchain data, you might find the new REST interface in the soon-to-be-released v0.10.0 to be more appropriate.

  -rest                  Accept public REST requests (default: 0)

[adamkrellenstein]

Alright then. Feel free to close this issue.

[miohtama]

Just for you information if you need thread-safe implementation of AuthServiceProxy, I wrote one using threading.local() and you can lift it off from here:

https://bitbucket.org/miohtama/cryptoassets/src/7693fea0cdfb0c0561071e3df15532d33c094bf7/cryptoassets/core/backend/bitcoind.py?at=master#cl-110