search

lbryio / lbry-sdk

The LBRY SDK for building decentralized, censorship resistant, monetized, digital content apps.
https://lbry.com
MIT License
7.19k stars 483 forks source link

transaction notification on clients sometimes doesn't happen until 2 new blocks come in #2945

Open tzarebczan opened 4 years ago

tzarebczan commented 4 years ago

Had it happen to me twice recently, even before the latest version. The last time, I think it waited for 2 blocks (may have published right around the time of the first one) for it to return from the blocking call. As soon as I saw the block come in, the publish finished.

We are also seeing timeouts on lbry.tv which could be related to this.

tzarebczan commented 4 years ago

Better, but still an issue - 2 blocks came in before it accepted the tx. Client side? Server side? Was connected to 3.209.188.122:50001


2020-05-15 11:19:23,036 INFO     lbry.extras.daemon.daemon:680: stream_abandon () {'txid': '3fdbeb5565a6b065d235ff0aafd880d8d71efc721e761ab24ede1fa0df82644f', 'nout': 0, 'blocking': True}
2020-05-15 11:19:23,123 INFO     lbry.extras.daemon.daemon:2113: Deleted file 1995.199.16.b_full.tif
2020-05-15 11:19:32,116 INFO     lbry.wallet.ledger:153: lbc_mainnet: added BlockHeightEvent(height=764890, change=1) header blocks, final height 764890
2020-05-15 11:23:27,429 INFO     lbry.wallet.ledger:153: lbc_mainnet: added BlockHeightEvent(height=764891, change=1) header blocks, final height 764891
2020-05-15 11:23:29,619 INFO     lbry.wallet.ledger:138: (lbc_mainnet) on_transaction: address=bQMwMTB6chWPi62T6Pbxd2E1Aw6XM96kNS, height=764891, is_verified=True, tx.id=d0f65a02339b000143e43ae4d8b9be95a074e44894bc9c7f6ac1d14719fdacf2
2020-05-15 11:23:29,631 INFO     lbry.wallet.ledger:138: (lbc_mainnet) on_transaction: address=bL73LKYofSDry9HSRedz9ztPaWGTZRxynQ, height=764891, is_verified=True, tx.id=d0f65a02339b000143e43ae4d8b9be95a074e44894bc9c7f6ac1d14719fdacf2```
tzarebczan commented 4 years ago

This should be fixed, but it seems to have caused a higher rate of missing input/mempool tx error rate.

tzarebczan commented 3 years ago

This is still an issue and happens when a tx is submitted when a new block comes in - it hangs until the next block is processed.

tzarebczan commented 3 years ago

Pleaseeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

lyoshenka commented 3 years ago

@shyba was looking at this recently

lyoshenka commented 3 years ago

this is a pretty big change. we'd need to change the mempool to use zmq, which is on the roadmap for the hub rewrite. after jack is done with his current work, this could be next on his list

tzarebczan commented 3 years ago

Still ran into it, but seems much harder to repro!

2021-07-22 16:22:53,206 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:53,207 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:54,205 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:54,205 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:54,206 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:55,208 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:55,208 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:55,209 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:56,200 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:56,201 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:56,201 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:57,205 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:57,205 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:57,206 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:58,211 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:58,211 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:58,212 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:58,496 INFO     lbry.wallet.ledger:150: lbc_mainnet: added BlockHeightEvent(height=999583, change=1) header blocks, final height 999583
2021-07-22 16:22:59,207 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
2021-07-22 16:22:59,208 WARNING  lbry.wallet.ledger:761: local history does not contain eb31d0ebdfb407a18ad08459e7df03064de2e6ac8ef4f07df9e54c3f11bf1bfc, requested height -1
tzarebczan commented 2 years ago

This still continues to cause issues on Odysee and YT sync

Some symptoms: publishes fail due to timeout, users try to reupload, causing duplicates tipping fails, causing duplicates since it's not clear/handled well YT sync doesn't record channel data since the call doesn't return within a few minutes.