Memory error in batch mode

[shivaenigma]

I am getting mempool tx from bitcoind in batch mode and sometimes getting this error:

File "/usr/local/lib/python2.7/dist-packages/bitcoinrpc/authproxy.py", line 164, in batch_ responses = self._get_response() File "/usr/local/lib/python2.7/dist-packages/bitcoinrpc/authproxy.py", line 186, in _get_response log.debug("<-- "+responsedata) MemoryError

Only getting sometimes, most of the times its working properly. Will try to reproduce exactly with a script

Note: harddisk and memory space is on box is normal

[shivaenigma]

Just FYI This coincided with the transaction surge yesterday https://www.reddit.com/r/Bitcoin/comments/3cdhtx/holy_smokes_50000_unconfirmed_transactions_and/

[shivaenigma]

Code to reproduce. Run this on mempool containing around 5000 tx and it starts to consume around 1gb memory and eventually crashes

from bitcoinrpc.authproxy import AuthServiceProxy
import sys
import traceback
import time
conn = AuthServiceProxy(sys.argv[1])
while(True):
    try:
      start_time = time.time()
      raw_mempool = conn.getrawmempool()
      cmd_list = [ ["getrawtransaction", tx_hash, 1] for tx_hash in raw_mempool]
      tx_list = conn.batch_(cmd_list)
      print "Got %d txs in %f sec"%(len(tx_list), time.time() - start_time)
    except Exception:
      traceback.print_exc()
      # Get a new connection
      conn = AuthServiceProxy(sys.argv[1])

[shivaenigma]

Closing this. Was resolved by decreasing size of batch. Fetching 1000 tx at a time.