freerko
commented
3 months ago Hi!
I've set up tapd 0.4.1-alpha with a local bitcoind 27.1 and try to sync the global default universe. I use the default SQLite and after about a week of syncing my tapd.db is only about 150 MB and I've only managed to sync a few assets, I think the stats didn't change since several days:
root@server:~# tapcli universe stats
{
"num_total_assets": "6440",
"num_total_groups": "37",
"num_total_syncs": "0",
"num_total_proofs": "6930"
}I get the "db tx retries exceeded" error as well:
2024-08-15 13:17:09.938 [DBG] UNIV: Looking up root node for base Universe (universe.Identifier) issuance-5cef8faf4ce2d667bb31ada76ff6d0a3c1d7944cf36a39f4af4d4e55084257a7
[... about 895 similar lines in the same second... ]
2024-08-15 13:17:09.941 [DBG] UNIV: Looking up root node for base Universe (universe.Identifier) issuance-8ab9ad693975feaabb386772753217a8d0122570610ca8a6aeec8ca900c745a0
2024-08-15 13:17:09.942 [WRN] UNIV: encountered an error whilst syncing with server=(universe.ServerAddr) {
ID: (int64) 1,
addrStr: (string) (len=32) "universe.lightning.finance:10029",
addr: (net.Addr) <nil>
}
: unable to register proofs: unable to register proofs: unable to register new group anchor issuance proofs: db tx retries exceeded
There is no high I/O on the local disk nor high CPU load. Is a global universe sync only supported/working with a Postgres DB? I've used SQLite for other projects with several gigabyte databases without issues. I have this issue on both testnet3 and mainnet.
guggero
jharveyb
dstadulis
Background
Describe your issue here.
When running with tapd v0.3.0 and a global sync of all issuances on, I get numerous log messages about DB TX retries due to serialization errors. In some cases, these operations hit the 10 retry limit and cause sync to fail.
I know that for 0.3.1 onwards global sync is not the default, but IMO this is an underlying issue that I think could appear even without global sync.
Your environment
Steps to reproduce
Tell us how to reproduce this issue. Please provide stacktraces and links to code in question.
Perform global issuance sync with the Lightning Labs default universe, with a fresh tapd.
From the log lines preceding these DB issues, I'm pretty sure this is triggered when multiple large (3000+ leaf) asset groups are being synced, and those proofs are being written to disk. The default batch size here is 200 elements, which can be a max size of at least 200 MiB. Since the sync is parallel, we can have multiple goroutines attempting writes in parallel.
The log lines aren't tagged with a goroutine ID or any way to distinguish writers, but based on the attempt counter it seems like 10+ concurrent writers.
Partial logs, generated with
cat tapd.log | rg serializ -C20:Possible fixes
I expect most end users with smaller wallets (not universe operators, companies running larger nodes) with run with the defaults and on an SQLite backend.
We could decrease the default DB write batch size, or the # of parallel goroutines during sync.
Alternatively, we could add another synchronization layer between concurrent DB writers like a counting semaphore to rate-limit DB access but still have many goroutines for the other work performed during sync.