search

vtnerd / monero-lws

Monero Light Wallet Server (scans monero viewkeys and implements mymonero API). Fast LMDB backend.
BSD 3-Clause "New" or "Revised" License
72 stars 27 forks source link

New accounts are 'pushed' to worker threads #102

Closed vtnerd closed 5 months ago

vtnerd commented 6 months ago

If a new account is added via cli or REST api, the status checking thread will "push" (via ZMQ) the new accounts to 1 of the scanning threads (chosen via fair queue). The current behavior is to reset all of the scanning threads. This should have better performance when lots of accounts are being added to a server.

I decided to use ZeroMQ instead of a lock with c++ queue so that this could be expanded into pushing accounts into other processes entirely. That effort is a difficult one for sure, but worthwhile if it can be achieved (I am not certain whether ZMQ can be used for that - or some other solution).

The account -> msgpack -> account step should be fairly quick, given that integer keys are used, etc.

vtnerd commented 6 months ago

Force pushed a rebase after some merging onto develop branch.

vtnerd commented 5 months ago

The account -> msgpack -> account step should be fairly quick, given that integer keys are used, etc.

I profiled the time to serialize an account with a testnet account with many transfers (mostly spends). It took ~280 microseconds to serialize and ~500 microseconds to de-serialize. So transferring accounts via ZMQ is indeed fairly quick.

With that said, this may change to a c++ queue with lock because ZMQ across processes just isn't going to work.

vtnerd commented 5 months ago

Force pushed another rebase.

vtnerd commented 5 months ago

I might change how this works slightly, as I think the worker threads should immediately register for PULL socket instead of waiting until middle of the scan loop.

vtnerd commented 5 months ago

Rebased again after recent merges.