search

PeerAssets / pypeerassets

Reference implementation of the PeerAssets protocol.
BSD 3-Clause "New" or "Revised" License
12 stars 16 forks source link

Rework entire card parsing subsystem #110

Closed peerchemist closed 5 years ago

peerchemist commented 6 years ago

find_all_valid cards should not return batches, but continual stream of cards.

Each of them should return generator.

saeveritt commented 6 years ago

Within find_all_valid_cards these are the most time consuming tasks

 Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
      557   87.584    0.157   87.584    0.157 {method 'acquire' of '_thread.lock' objects}
      145    0.142    0.001    0.142    0.001 {method 'recv_into' of '_socket.socket' objects}
      158    0.056    0.000    0.056    0.000 {method '_acquire_restore' of '_thread.RLock' objects}
        2    0.006    0.003    0.006    0.003 decoder.py:345(raw_decode)
      226    0.002    0.000   87.620    0.388 _base.py:378(result)
     1254    0.001    0.000   87.804    0.070 __main__.py:190(<genexpr>)
      160    0.001    0.000   87.641    0.548 threading.py:263(wait)
      922    0.001    0.000    0.001    0.000 {method 'decode' of 'bytes' objects}

The cumulative time spent for acquiring the _thread.lock objects is due to the rpc calls within each of them. I propose we iterate through the cards, store the blockhashes in a set, then batch rpc call the hashes rather than doing them individually.

peerchemist commented 5 years ago

Solved with https://github.com/PeerAssets/pypeerassets/pull/126

peerchemist commented 5 years ago

@saeveritt can you do that time test again?