search

lbryio / lbry-sdk

The LBRY SDK for building decentralized, censorship resistant, monetized, digital content apps.
https://lbry.com
MIT License
7.19k stars 482 forks source link

test_spec_example: send_to_address_and_wait(address, 400.0) hangs test #3666

Open moodyjon opened 2 years ago

moodyjon commented 2 years ago

Similar to lbryio/hub#98, but no reconnect is involved. Extra hub tracing for notifications in moodyjon/hub was NOT enabled, so we don't see much detail.

https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405

3514
test_spec_example (integration.takeovers.test_resolve_command.ResolveClaimTakeovers) ... 2022-09-15 18:56:41,767 - asyncio - WARNING - Executing <Task pending coro=<CommandTestCase.asyncSetUp() running at /home/runner/work/lbry-sdk/lbry-sdk/lbry/testcase.py:415> wait_for=<Future pending cb=[_chain_future.<locals>._call_check_cancel() at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/futures.py:351, <TaskWakeupMethWrapper object at 0x7f654c4b6910>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> cb=[_run_until_complete_cb() at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:157] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:566> took 1.012 seconds
[3515](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3516)
2022-09-15 18:56:43,748 - elasticsearch - WARNING - GET http://localhost:9200/_template/9c608281e815443fb014651050bc2a4dclaims [status:404 request:0.002s]
[3516](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3517)
2022-09-15 18:56:45,324 - lbry.wallet.network - WARNING - wait reconnect+keepalive
[3517](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3518)
2022-09-15 18:56:45,419 - lbry.wallet.ledger - WARNING - initial status update address mqvGxpeAsSusqCGMe3jQRx6Wwcgq6BKaVY status None
[3518](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3519)
2022-09-15 18:56:45,419 - lbry.wallet.ledger - WARNING - initial status update address n2TPX6skMYygny9ivjqFF7QVGcS94GU8Xr status None
[3519](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3520)
2022-09-15 18:56:45,419 - lbry.wallet.ledger - WARNING - initial status update address mpmwDXYVYRNYzaRpUZM3rK2CD76icHdiGb status None
[3520](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3521)
2022-09-15 18:56:45,420 - lbry.wallet.ledger - WARNING - initial status update address mmyksKrFhekEQZa4LNPaFAAJKJ1WNM9nZH status None
[3521](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3522)
2022-09-15 18:56:45,420 - lbry.wallet.ledger - WARNING - initial status update address mmtZg7ni6RAKNqbnBcusPRrzFwnNZ5fp4Y status None
[3522](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3523)
2022-09-15 18:56:45,420 - lbry.wallet.ledger - WARNING - initial status update address moGtYHjqQJ3nHhMtJ6x2eiKwsENut92yV8 status None
[3523](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3524)
2022-09-15 18:56:45,421 - lbry.wallet.ledger - WARNING - initial status update address mq38kPcMgw3uGVeBfYCZSbA5xDBEp4LyD1 status None
[3524](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3525)
2022-09-15 18:56:45,421 - lbry.wallet.ledger - WARNING - initial status update address mhC5dDJmskpdoTYBRcrHZENTCQszuzgc7Y status None
[3525](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3526)
2022-09-15 18:56:45,421 - lbry.wallet.ledger - WARNING - initial status update address mwqB8bKUwcsjXTxmpEMk4f675C7J6F2JvL status None
[3526](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3527)
2022-09-15 18:56:45,422 - lbry.wallet.ledger - WARNING - initial status update address n39GG9ej6emTvY5AmdNxvXxBh1UR1NSvCm status None
[3527](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3528)
2022-09-15 18:56:45,422 - lbry.wallet.ledger - WARNING - initial status update address msaUzYxnQcsB3LzTJYFRDciY7JKo1R21iv status None
[3528](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3529)
2022-09-15 18:56:45,422 - lbry.wallet.ledger - WARNING - initial status update address mi2CJScakJVpS7d4AJt7z8r3V5HxKN2A2x status None
[3529](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3530)
2022-09-15 18:56:45,423 - lbry.wallet.ledger - WARNING - initial status update address mxsfuRsgXRZQCby7rR6AKsvB8JPJXPLrW8 status None
[3530](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3531)
2022-09-15 18:56:45,423 - lbry.wallet.ledger - WARNING - initial status update address mp6xvdVZdERXaticupeVvzqTxZWP6rr8rN status None
[3531](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3532)
2022-09-15 18:56:45,423 - lbry.wallet.ledger - WARNING - initial status update address myVY7t9RNWbYtNnHXVqFZjZSAycJXs4W9w status None
[3532](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3533)
2022-09-15 18:56:45,424 - lbry.wallet.ledger - WARNING - initial status update address mshLiHt2WLGyTBHPwyMq6RELLPm31ouFuF status None
[3533](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3534)
2022-09-15 18:56:45,424 - lbry.wallet.ledger - WARNING - initial status update address mzy1xwMiP82LLSoGzfSmmbaqAUk4EMZzAV status None
[3534](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3535)
2022-09-15 18:56:45,424 - lbry.wallet.ledger - WARNING - initial status update address n3WLFA5kwyVE9gYUVXwCbyRQq4BWJPbKfA status None
[3535](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3536)
2022-09-15 18:56:45,425 - lbry.wallet.ledger - WARNING - initial status update address mx3jGTAf6ngYLdTMWTF1bwmhdhnqiW9rZD status None
[3536](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3537)
2022-09-15 18:56:45,425 - lbry.wallet.ledger - WARNING - initial status update address mqvEwJxEdXDB4LGXhF4CQwT6ZkQfmvFcnW status None
[3537](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3538)
2022-09-15 18:56:45,480 - lbry.wallet.ledger - WARNING - initial status update address miWuDnqCRa6T3Xkpwy1GeNtcWxwYwtcvsD status None
[3538](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3539)
2022-09-15 18:56:45,480 - lbry.wallet.ledger - WARNING - initial status update address n3ZWyoczQNDnGfzk8Z2jAuf9uVrM79jvs5 status None
[3539](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3540)
2022-09-15 18:56:45,480 - lbry.wallet.ledger - WARNING - initial status update address n2gBXorN4AK5tE3HAxxfJmzGQ4Ms9YimE1 status None
[3540](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3541)
2022-09-15 18:56:45,481 - lbry.wallet.ledger - WARNING - initial status update address miEPidCpivRQ5W7mSbHXh2YZxeEhD5C36D status None
[3541](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3542)
2022-09-15 18:56:45,481 - lbry.wallet.ledger - WARNING - initial status update address mkJwZSXw9VqjH1fNVyp2JqoP3nFpjMAugE status None
[3542](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3543)
2022-09-15 18:56:45,481 - lbry.wallet.ledger - WARNING - initial status update address n1Cesg9NRCDZ8ipAMQuzW9ojYeHeCsEQ4o status None
[3543](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3544)
2022-09-15 18:56:45,707 - lbry.testcase - WARNING - before: done = False, tx_watch = []
[3544](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3545)
2022-09-15 18:56:45,757 - lbry.wallet.ledger - WARNING - status update address mqvGxpeAsSusqCGMe3jQRx6Wwcgq6BKaVY status 30e663346d037f2ceb2e50ccfa6d1975fe625841941e2575408aab746e0a0a99
[3545](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3546)
2022-09-15 18:56:45,774 - lbry.testcase - WARNING - after: done = False, tx_watch = []
[3546](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3547)
2022-09-15 18:56:45,782 - lbry.wallet.ledger - WARNING - updated address: mqvGxpeAsSusqCGMe3jQRx6Wwcgq6BKaVY history: b0f187bf634d0b7dd8f2beb85aff1188313f229bc5ceddac569281e332507c98:0:
[3547](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3548)
2022-09-15 18:56:45,814 - lbry.wallet.ledger - WARNING - initial status update address moCd6v5VZT3HzpbeQ2X61epmk3EkR51GYB status None
[3548](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3549)
2022-09-15 18:56:45,960 - lbry.wallet.ledger - WARNING - status update address mqvGxpeAsSusqCGMe3jQRx6Wwcgq6BKaVY status 6d886384f7604f1a6d325e8febb3b047a180d9995db5491ca80a19ae100374ea
[3549](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3550)
2022-09-15 18:56:45,990 - lbry.wallet.ledger - WARNING - updated address: mqvGxpeAsSusqCGMe3jQRx6Wwcgq6BKaVY history: b0f187bf634d0b7dd8f2beb85aff1188313f229bc5ceddac569281e332507c98:201:
[3550](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3551)
2022-09-15 18:56:53,322 - lbry.testcase - WARNING - before: done = False, tx_watch = []
[3551](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3552)
Stack for <Task pending coro=<Event.wait() running at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/locks.py:293> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654c64b9d0>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> cb=[_wait.<locals>._on_completion() at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/tasks.py:466] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/tasks.py:387> (most recent call last):
[3552](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3553)
Stack for <Task pending coro=<ConnectionManager._update() running at /home/runner/work/lbry-sdk/lbry-sdk/lbry/connection_manager.py:70> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654ca8a2d0>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> created at /home/runner/work/lbry-sdk/lbry-sdk/lbry/connection_manager.py:105> (most recent call last):
[3553](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3554)
Stack for <Task pending coro=<DiskSpaceManager.cleaning_loop() running at /home/runner/work/lbry-sdk/lbry-sdk/lbry/blob/disk_space_manager.py:66> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654c5f2310>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> cb=[DiskSpaceManager.start.<locals>.<lambda>() at /home/runner/work/lbry-sdk/lbry-sdk/lbry/blob/disk_space_manager.py:72] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/tasks.py:351> (most recent call last):
[3554](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3555)
Stack for <Task pending coro=<BlobServer.start_server.<locals>._start_server() running at /home/runner/work/lbry-sdk/lbry-sdk/lbry/blob_exchange/server.py:186> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654c4e3510>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> created at /home/runner/work/lbry-sdk/lbry-sdk/lbry/blob_exchange/server.py:188> (most recent call last):
[3555](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3556)
Stack for <Task pending coro=<SessionManager.serve() running at /home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/herald/session.py:599> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654c5e1e90>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> created at /home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/service.py:58> (most recent call last):
[3556](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3557)
Stack for <Task pending coro=<ElasticNotifierClientProtocol.maintain_connection() running at /home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/notifier_protocol.py:54> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654cff1650>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> created at /home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/service.py:58> (most recent call last):
[3557](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3558)
Stack for <Task pending coro=<SessionManager._clear_stale_sessions() running at /home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/herald/session.py:358> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654c418c10>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> cb=[_wait.<locals>._on_completion() at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/tasks.py:466] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/tasks.py:387> (most recent call last):
[3558](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3559)
Stack for <Task pending coro=<ClientSession.keepalive_loop() running at /home/runner/work/lbry-sdk/lbry-sdk/lbry/wallet/network.py:127> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654c548890>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> cb=[_wait.<locals>._on_completion() at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/tasks.py:466] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/tasks.py:351> (most recent call last):
[3559](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3560)
Stack for <Task pending coro=<ElasticSyncService.run_es_notifier() running at /home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/elastic_sync/service.py:65> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654c92a550>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> created at /home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/service.py:58> (most recent call last):
[3560](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3561)
Stack for <Task pending coro=<HubMemPool.send_notifications_forever() running at /home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/herald/mempool.py:236> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654c4895d0>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> created at /home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/service.py:58> (most recent call last):
[3561](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3562)
Stack for <Task pending coro=<ReflectorServer.start_server.<locals>._start_server() running at /home/runner/work/lbry-sdk/lbry-sdk/lbry/stream/reflector/server.py:189> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654caf6850>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> created at /home/runner/work/lbry-sdk/lbry-sdk/lbry/stream/reflector/server.py:193> (most recent call last):
[3562](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3563)
Stack for <Task pending coro=<SessionManager._manage_servers() running at /home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/herald/session.py:304> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654c56a0d0>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> cb=[_wait.<locals>._on_completion() at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/tasks.py:466] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/tasks.py:387> (most recent call last):
[3563](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3564)
Stack for <Task pending coro=<AnalyticsManager.run() running at /home/runner/work/lbry-sdk/lbry-sdk/lbry/extras/daemon/analytics.py:138> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654c498090>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/tasks.py:351> (most recent call last):
[3564](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3565)
Stack for <Task pending coro=<BlockchainProcessorService.process_blocks_and_mempool_forever() running at /home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/scribe/service.py:1781> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654c5d79d0>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> created at /home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/service.py:58> (most recent call last):
[3565](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3566)
Stack for <Task pending coro=<BlockchainReaderService.refresh_blocks_forever() running at /home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/service.py:261> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654f9e5ed0>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> created at /home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/service.py:58> (most recent call last):
[3566](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3567)
Stack for <Task pending coro=<Network.network_loop() running at /home/runner/work/lbry-sdk/lbry-sdk/lbry/wallet/network.py:351> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654c7093d0>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> cb=[Network.start.<locals>.loop_task_done_callback() at /home/runner/work/lbry-sdk/lbry-sdk/lbry/wallet/network.py:222] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/tasks.py:351> (most recent call last):
[3567](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3568)
Stack for <Task pending coro=<RPCSession._receive_messages() running at /home/runner/work/lbry-sdk/lbry-sdk/lbry/wallet/rpc/session.py:406> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f654c6e70d0>()] created at /opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py:395> created at /home/runner/work/lbry-sdk/lbry-sdk/lbry/wallet/rpc/session.py:181> (most recent call last):  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/locks.py", line 293, in wait
[3568](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3569)
    await fut
[3569](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3570)
  File "/home/runner/work/lbry-sdk/lbry-sdk/lbry/connection_manager.py", line 70, in _update
[3570](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3571)
    await asyncio.sleep(0.1, loop=self.loop)
[3571](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3572)
  File "/home/runner/work/lbry-sdk/lbry-sdk/lbry/blob/disk_space_manager.py", line 66, in cleaning_loop
[3572](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3573)
    await asyncio.sleep(self.cleaning_interval)
[3573](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3574)
  File "/home/runner/work/lbry-sdk/lbry-sdk/lbry/blob_exchange/server.py", line 186, in _start_server
[3574](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3575)
    await server.serve_forever()
[3575](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3576)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/herald/session.py", line 599, in serve
[3576](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3577)
    self._manage_servers()
[3577](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3578)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/notifier_protocol.py", line 54, in maintain_connection
[3578](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3579)
    await self._lost_connection.wait()
[3579](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3580)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/herald/session.py", line 358, in _clear_stale_sessions
[3580](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3581)
    await sleep(session_timeout // 10)
[3581](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3582)
  File "/home/runner/work/lbry-sdk/lbry-sdk/lbry/wallet/network.py", line 127, in keepalive_loop
[3582](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3583)
    await asyncio.sleep(max(0, max_idle - (now - self.last_send)))
[3583](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3584)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/elastic_sync/service.py", line 65, in run_es_notifier
[3584](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3585)
    await server.serve_forever()
[3585](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3586)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/herald/mempool.py", line 236, in send_notifications_forever
[3586](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3587)
    (session_id, height_changed, hashXes) = await self._notification_q.get()
[3587](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3588)
  File "/home/runner/work/lbry-sdk/lbry-sdk/lbry/stream/reflector/server.py", line 189, in _start_server
[3588](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3589)
    await server.serve_forever()
[3589](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3590)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/herald/session.py", line 304, in _manage_servers
[3590](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3591)
    await self.session_event.wait()
[3591](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3592)
  File "/home/runner/work/lbry-sdk/lbry-sdk/lbry/extras/daemon/analytics.py", line 138, in run
[3592](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3593)
    await asyncio.sleep(1800)
[3593](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3594)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/scribe/service.py", line 1781, in process_blocks_and_mempool_forever
[3594](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3595)
    await asyncio.wait_for(self.blocks_event.wait(), self.wait_for_blocks_duration)
[3595](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3596)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/service.py", line 261, in refresh_blocks_forever
[3596](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3597)
    await asyncio.sleep(self._refresh_interval)
[3597](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3598)
  File "/home/runner/work/lbry-sdk/lbry-sdk/lbry/wallet/network.py", line 351, in network_loop
[3598](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3599)
    return_when=asyncio.FIRST_COMPLETED
[3599](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3600)
  File "/home/runner/work/lbry-sdk/lbry-sdk/lbry/wallet/rpc/session.py", line 406, in _receive_messages
[3600](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3601)
    message = await self.framer.receive_message()
[3601](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3602)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/service.py", line 261, in refresh_blocks_forever
[3602](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3603)
    await asyncio.sleep(self._refresh_interval)
[3603](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3604)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/herald/service.py", line 110, in receive_es_notifications
[3604](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3605)
    self._es_height, self._es_block_hash = await self.es_notifications.get()
[3605](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3606)
  File "/home/runner/work/lbry-sdk/lbry-sdk/lbry/stream/stream_manager.py", line 167, in reflect_streams
[3606](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3607)
    return await self._reflect_streams()
[3607](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3608)
  File "/home/runner/work/lbry-sdk/lbry-sdk/lbry/extras/daemon/components.py", line 744, in announce_forever
[3608](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3609)
    await asyncio.sleep(sleep_seconds)
[3609](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3610)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/scribe/prefetcher.py", line 47, in main_loop
[3610](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3611)
    await asyncio.sleep(self.polling_delay)
[3611](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3612)
  File "/home/runner/work/lbry-sdk/lbry-sdk/.tox/takeovers/lib/python3.7/site-packages/hub/herald/session.py", line 1033, in _receive_messages
[3612](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3613)
    message = await self.framer.receive_message()
[3613](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3614)
  File "/home/runner/work/lbry-sdk/lbry-sdk/tests/integration/takeovers/test_resolve_command.py", line 1067, in test_spec_example
[3614](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3615)
    await self.send_to_address_and_wait(address, 400.0)
[3615](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3616)
  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/locks.py", line 293, in wait
[3616](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3617)
    await fut
[3617](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:3618)
ERROR
======================================================================
[5210](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:5211)
ERROR: test_spec_example (integration.takeovers.test_resolve_command.ResolveClaimTakeovers)
[5211](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:5212)
----------------------------------------------------------------------
[5212](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:5213)
Traceback (most recent call last):
[5213](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:5214)
  File "/home/runner/work/lbry-sdk/lbry-sdk/lbry/testcase.py", line 145, in run
[5214](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:5215)
    self.loop.run_until_complete(maybe_coroutine)
[5215](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:5216)
  File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
[5216](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:5217)
    return future.result()
[5217](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:5218)
concurrent.futures._base.CancelledError
[5218](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:5219)

[5219](https://github.com/moodyjon/lbry-sdk/actions/runs/3062810249/jobs/4944196405#step:11:5220)
----------------------------------------------------------------------

Test code:

        name = 'test'
        await self.generate(494)
        address = (await self.account.receiving.get_addresses(True))[0]
        await self.send_to_address_and_wait(address, 400.0).     <<<<<<<<<<< HANG here
        await self.account.ledger.on_address.first
moodyjon commented 2 years ago

Likely fixed by https://github.com/lbryio/hub/pull/99

Will monitor for failures after hub version bump and then close.

moodyjon commented 1 year ago

Seen again AFTER all the fixes in https://github.com/lbryio/lbry-sdk/pull/3707: https://github.com/moodyjon/lbry-sdk/actions/runs/3515119954/jobs/5889980919