search

lbryio / lbry-sdk

The LBRY SDK for building decentralized, censorship resistant, monetized, digital content apps.
https://lbry.com
MIT License
7.19k stars 482 forks source link

Connection reset from API when using Python 3.8 #2769

Open mkroman opened 4 years ago

mkroman commented 4 years ago

After some testing and some discussion with @er013 it's apparent that there's some kind of silent incompatibility with Python 3.8.x that's not present in Python 3.7.6.

When lbrynet (0.58.1) is started using Python 3.8.1, requests to the API will be met with a connection reset:

% python -V
Python 3.8.1
% lbrynet start    
2020-02-06 03:49:21,641 INFO     lbry.extras.daemon.daemon:450: Starting LBRYNet Daemon
2020-02-06 03:49:21,642 INFO     lbry.extras.daemon.daemon:452: Platform: {
  "processor": "",
  "python_version": "3.8.1",
  "platform": "Linux-5.5.2-arch1-1-x86_64-with-glibc2.29",
  "os_release": "5.5.2-arch1-1",
  "os_system": "Linux",
  "lbrynet_version": "0.58.1",
  "version": "0.58.1",
  "build": "dev",
  "distro": {
    "id": "arch",
    "version": "rolling",
    "version_parts": {
      "major": "",
      "minor": "",
      "build_number": ""
    },
    "like": "",
    "codename": "n/a"
  },
  "desktop": "Unknown"
}
2020-02-06 03:49:21,654 INFO     lbry.extras.daemon.daemon:461: RPC server listening on TCP ::1:5279
2020-02-06 03:49:21,654 INFO     lbry.extras.daemon.daemon:471: media server listening on TCP 127.0.0.1:5280
2020-02-06 03:49:22,200 INFO     lbry.extras.daemon.exchange_rate_manager:184: Starting exchange rate manager
2020-02-06 03:49:22,201 INFO     lbry.extras.daemon.components:457: detecting external ip
2020-02-06 03:49:22,203 INFO     lbry.extras.daemon.components:65: Loading databases
2020-02-06 03:49:22,203 INFO     lbry.wallet.database:238: connecting to database: /home/mk/.local/share/lbry/lbrynet/lbrynet.sqlite
2020-02-06 03:49:22,207 WARNING  lbry.extras.daemon.components:387: upnp discovery failed: 
2020-02-06 03:49:22,208 ERROR    lbry.extras.daemon.components:472: failed to setup upnp
2020-02-06 03:49:22,657 INFO     lbry.extras.daemon.components:476: detected external ip using lbry.com fallback
2020-02-06 03:49:22,658 INFO     lbry.extras.daemon.components:232: start the dht
2020-02-06 03:49:22,659 INFO     lbry.extras.daemon.components:256: Started the dht
2020-02-06 03:49:22,660 INFO     lbry.extras.daemon.components:154: Starting wallet
2020-02-06 03:49:23,082 INFO     lbry.wallet.database:238: connecting to database: /home/mk/.local/share/lbry/lbryum/lbc_mainnet/blockchain.db
2020-02-06 03:49:23,083 INFO     lbry.dht.node:126: DHT node listening on UDP 0.0.0.0:4444
2020-02-06 03:49:23,244 WARNING  lbry.wallet.network:73: Wallet server (spv8.lbry.com:50001) returned an error. Code: 1 Message: unsupported protocol version: 0.58.1
2020-02-06 03:49:23,295 WARNING  lbry.wallet.network:73: Wallet server (spv8.lbry.com:50001) returned an error. Code: 1 Message: unsupported protocol version: 0.58.1
2020-02-06 03:49:23,412 INFO     lbry.wallet.network:342: connected to spv9.lbry.com:50001
2020-02-06 03:49:23,413 INFO     lbry.wallet.network:187: Switching to SPV wallet server: spv9.lbry.com:50001
2020-02-06 03:49:23,520 INFO     lbry.wallet.network:191: Subscribed to headers: spv9.lbry.com:50001
2020-02-06 03:49:23,520 INFO     lbry.wallet.ledger:315: Subscribing and updating accounts.
2020-02-06 03:49:23,627 INFO     lbry.wallet.ledger:428: Subscribe to 5 accounts
2020-02-06 03:49:23,736 INFO     lbry.wallet.ledger:457: subscribed to 1/1 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,736 INFO     lbry.wallet.ledger:459: finished subscribing to 1 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,738 INFO     lbry.wallet.ledger:457: subscribed to 6/6 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,738 INFO     lbry.wallet.ledger:459: finished subscribing to 6 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,745 INFO     lbry.wallet.ledger:457: subscribed to 37/37 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,746 INFO     lbry.wallet.ledger:459: finished subscribing to 37 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,826 INFO     lbry.wallet.ledger:457: subscribed to 33/33 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,826 INFO     lbry.wallet.ledger:459: finished subscribing to 33 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,827 INFO     lbry.wallet.ledger:457: subscribed to 22/22 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,827 INFO     lbry.wallet.ledger:459: finished subscribing to 22 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,850 INFO     lbry.wallet.ledger:457: subscribed to 20/20 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,850 INFO     lbry.wallet.ledger:459: finished subscribing to 20 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,941 INFO     lbry.wallet.ledger:457: subscribed to 21/21 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,941 INFO     lbry.wallet.ledger:459: finished subscribing to 21 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,945 INFO     lbry.wallet.ledger:457: subscribed to 8/8 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,945 INFO     lbry.wallet.ledger:459: finished subscribing to 8 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,955 INFO     lbry.wallet.ledger:457: subscribed to 61/61 addresses on 216.19.244.55:50001
2020-02-06 03:49:23,955 INFO     lbry.wallet.ledger:459: finished subscribing to 61 addresses on 216.19.244.55:50001
2020-02-06 03:49:24,317 INFO     lbry.wallet.ledger:693: Loaded account bHrXsS75NUZ6sBikUtBuS4Z1sdsgFhsjp2 with 0.0 LBC, 20 receiving addresses (gap: 20), 6 change addresses (gap: 6), 0 channels, 0 certificates and 0 claims. 
2020-02-06 03:49:24,319 INFO     lbry.wallet.ledger:693: Loaded account bGQ5q7Rq36H6gzZyKvn58stCHE2AXKpgkY with 17.652122 LBC, 37 receiving addresses (gap: 20), 21 change addresses (gap: 6), 0 channels, 0 certificates and 3 claims. 
2020-02-06 03:49:24,322 INFO     lbry.wallet.ledger:693: Loaded account bCnxamLxj8n84abYgNRKsjAawFfW3NGYYk with 114.7994955 LBC, 61 receiving addresses (gap: 20), 22 change addresses (gap: 6), 4 channels, 4 certificates and 7 claims. 
2020-02-06 03:49:24,322 INFO     lbry.wallet.ledger:687: Loaded single key account bQb54gW4cK8dbjBW8n4oewTUsXJU41TRbD with 25.0 LBC. 0 channels, 0 certificates and 0 claims
2020-02-06 03:49:24,324 INFO     lbry.wallet.ledger:693: Loaded account bJhGRNmFiNUdh4Ahy4K73brm4EpaDqjaEz with 44.1 LBC, 33 receiving addresses (gap: 20), 8 change addresses (gap: 6), 0 channels, 0 certificates and 1 claims. 
2020-02-06 03:49:24,324 INFO     lbry.extras.daemon.components:341: start blob server
2020-02-06 03:49:24,324 INFO     lbry.extras.daemon.components:279: Started blob announcer
2020-02-06 03:49:24,325 INFO     lbry.extras.daemon.components:316: Starting the file manager
2020-02-06 03:49:24,326 INFO     lbry.stream.stream_manager:150: Initializing 0 files
2020-02-06 03:49:24,326 INFO     lbry.stream.stream_manager:166: Started stream manager with 0 files
2020-02-06 03:49:24,326 INFO     lbry.extras.daemon.components:322: Done setting up file manager
2020-02-06 03:49:24,327 INFO     lbry.blob_exchange.server:176: Blob server listening on TCP 0.0.0.0:3333
2020-02-06 03:49:26,230 INFO     lbry.dht.protocol.routing_table:289: join buckets 2
2020-02-06 03:49:26,230 INFO     lbry.dht.protocol.routing_table:289: join buckets 1
% lbrynet version
Could not connect to daemon. Are you sure it's running?
% curl -vv 127.0.0.1:5279
*   Trying 127.0.0.1:5279...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 5279 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:5279
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

However, if the same lbrynet version (0.58.1) is installed with Python 3.7.6 the API will respond successfully:


% python -V
Python 3.7.6
% lbrynet start
2020-02-06 03:50:47,386 INFO     lbry.extras.daemon.daemon:450: Starting LBRYNet Daemon
2020-02-06 03:50:47,387 INFO     lbry.extras.daemon.daemon:452: Platform: {
  "processor": "",
  "python_version": "3.7.6",
  "platform": "Linux-5.5.2-arch1-1-x86_64-with-arch-Arch-Linux",
  "os_release": "5.5.2-arch1-1",
  "os_system": "Linux",
  "lbrynet_version": "0.58.1",
  "version": "0.58.1",
  "build": "dev",
  "distro": {
    "id": "arch",
    "version": "rolling",
    "version_parts": {
      "major": "",
      "minor": "",
      "build_number": ""
    },
    "like": "",
    "codename": "n/a"
  },
  "desktop": "Unknown"
}
2020-02-06 03:50:47,400 INFO     lbry.extras.daemon.daemon:461: RPC server listening on TCP ::1:5279
2020-02-06 03:50:47,401 INFO     lbry.extras.daemon.daemon:471: media server listening on TCP 127.0.0.1:5280
2020-02-06 03:50:47,838 INFO     lbry.extras.daemon.components:65: Loading databases
2020-02-06 03:50:47,838 INFO     lbry.wallet.database:238: connecting to database: /home/mk/.local/share/lbry/lbrynet/lbrynet.sqlite
2020-02-06 03:50:47,839 INFO     lbry.extras.daemon.exchange_rate_manager:184: Starting exchange rate manager
2020-02-06 03:50:47,839 INFO     lbry.extras.daemon.components:457: detecting external ip
2020-02-06 03:50:47,898 WARNING  lbry.extras.daemon.components:387: upnp discovery failed: list index out of range
2020-02-06 03:50:47,898 ERROR    lbry.extras.daemon.components:472: failed to setup upnp
2020-02-06 03:50:48,355 INFO     lbry.extras.daemon.components:476: detected external ip using lbry.com fallback
2020-02-06 03:50:48,357 INFO     lbry.extras.daemon.components:154: Starting wallet
2020-02-06 03:50:48,759 INFO     lbry.extras.daemon.components:232: start the dht
2020-02-06 03:50:48,759 INFO     lbry.extras.daemon.components:256: Started the dht
2020-02-06 03:50:48,760 INFO     lbry.wallet.database:238: connecting to database: /home/mk/.local/share/lbry/lbryum/lbc_mainnet/blockchain.db
2020-02-06 03:50:48,761 INFO     lbry.dht.node:126: DHT node listening on UDP 0.0.0.0:4444
2020-02-06 03:50:48,859 WARNING  lbry.wallet.network:74: Wallet server (spv8.lbry.com:50001) returned an error. Code: 1 Message: unsupported protocol version: 0.58.1
2020-02-06 03:50:48,910 WARNING  lbry.wallet.network:74: Wallet server (spv8.lbry.com:50001) returned an error. Code: 1 Message: unsupported protocol version: 0.58.1
2020-02-06 03:50:49,032 INFO     lbry.wallet.network:342: connected to spv9.lbry.com:50001
2020-02-06 03:50:49,032 INFO     lbry.wallet.network:187: Switching to SPV wallet server: spv9.lbry.com:50001
2020-02-06 03:50:49,140 INFO     lbry.wallet.network:191: Subscribed to headers: spv9.lbry.com:50001
2020-02-06 03:50:49,141 INFO     lbry.wallet.ledger:315: Subscribing and updating accounts.
2020-02-06 03:50:49,248 INFO     lbry.wallet.ledger:428: Subscribe to 5 accounts
2020-02-06 03:50:49,357 INFO     lbry.wallet.ledger:458: subscribed to 20/20 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,357 INFO     lbry.wallet.ledger:461: finished subscribing to 20 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,361 INFO     lbry.wallet.ledger:458: subscribed to 37/37 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,361 INFO     lbry.wallet.ledger:461: finished subscribing to 37 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,434 INFO     lbry.wallet.ledger:458: subscribed to 61/61 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,434 INFO     lbry.wallet.ledger:461: finished subscribing to 61 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,435 INFO     lbry.wallet.ledger:458: subscribed to 8/8 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,435 INFO     lbry.wallet.ledger:461: finished subscribing to 8 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,435 INFO     lbry.wallet.ledger:458: subscribed to 1/1 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,436 INFO     lbry.wallet.ledger:461: finished subscribing to 1 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,466 INFO     lbry.wallet.ledger:458: subscribed to 6/6 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,467 INFO     lbry.wallet.ledger:461: finished subscribing to 6 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,551 INFO     lbry.wallet.ledger:458: subscribed to 21/21 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,551 INFO     lbry.wallet.ledger:461: finished subscribing to 21 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,557 INFO     lbry.wallet.ledger:458: subscribed to 22/22 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,557 INFO     lbry.wallet.ledger:461: finished subscribing to 22 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,558 INFO     lbry.wallet.ledger:458: subscribed to 33/33 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,558 INFO     lbry.wallet.ledger:461: finished subscribing to 33 addresses on 216.19.244.55:50001
2020-02-06 03:50:49,891 INFO     lbry.wallet.ledger:696: Loaded account bHrXsS75NUZ6sBikUtBuS4Z1sdsgFhsjp2 with 0.0 LBC, 20 receiving addresses (gap: 20), 6 change addresses (gap: 6), 0 channels, 0 certificates and 0 claims. 
2020-02-06 03:50:49,893 INFO     lbry.wallet.ledger:696: Loaded account bGQ5q7Rq36H6gzZyKvn58stCHE2AXKpgkY with 17.652122 LBC, 37 receiving addresses (gap: 20), 21 change addresses (gap: 6), 0 channels, 0 certificates and 3 claims. 
2020-02-06 03:50:49,895 INFO     lbry.wallet.ledger:696: Loaded account bCnxamLxj8n84abYgNRKsjAawFfW3NGYYk with 114.7994955 LBC, 61 receiving addresses (gap: 20), 22 change addresses (gap: 6), 4 channels, 4 certificates and 7 claims. 
2020-02-06 03:50:49,896 INFO     lbry.wallet.ledger:689: Loaded single key account bQb54gW4cK8dbjBW8n4oewTUsXJU41TRbD with 25.0 LBC. 0 channels, 0 certificates and 0 claims
2020-02-06 03:50:49,897 INFO     lbry.wallet.ledger:696: Loaded account bJhGRNmFiNUdh4Ahy4K73brm4EpaDqjaEz with 44.1 LBC, 33 receiving addresses (gap: 20), 8 change addresses (gap: 6), 0 channels, 0 certificates and 1 claims. 
2020-02-06 03:50:49,898 INFO     lbry.extras.daemon.components:279: Started blob announcer
2020-02-06 03:50:49,898 INFO     lbry.extras.daemon.components:341: start blob server
2020-02-06 03:50:49,898 INFO     lbry.extras.daemon.components:316: Starting the file manager
2020-02-06 03:50:49,899 INFO     lbry.blob_exchange.server:176: Blob server listening on TCP 0.0.0.0:3333
2020-02-06 03:50:49,899 INFO     lbry.stream.stream_manager:150: Initializing 0 files
2020-02-06 03:50:49,899 INFO     lbry.stream.stream_manager:166: Started stream manager with 0 files
2020-02-06 03:50:49,899 INFO     lbry.extras.daemon.components:322: Done setting up file manager
2020-02-06 03:50:51,809 INFO     lbry.dht.protocol.routing_table:289: join buckets 1
2020-02-06 03:50:54,137 INFO     lbry.dht.protocol.routing_table:289: join buckets 1
% lbrynet version 
{
  "build": "dev",
  "desktop": "Unknown",
  "distro": {
    "codename": "n/a",
    "id": "arch",
    "like": "",
    "version": "rolling",
    "version_parts": {
      "build_number": "",
      "major": "",
      "minor": ""
    }
  },
  "lbrynet_version": "0.58.1",
  "os_release": "5.5.2-arch1-1",
  "os_system": "Linux",
  "platform": "Linux-5.5.2-arch1-1-x86_64-with-arch-Arch-Linux",
  "processor": "",
  "python_version": "3.7.6",
  "version": "0.58.1"
}
% curl -vv 127.0.0.1:5279
*   Trying 127.0.0.1:5279...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 5279 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:5279
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 405 Method Not Allowed
< Content-Type: text/plain; charset=utf-8
< Allow: POST
< Content-Length: 23
< Date: Thu, 06 Feb 2020 03:16:38 GMT
< Server: Python/3.7 aiohttp/3.5.4
< 
* Connection #0 to host 127.0.0.1 left intact
405: Method Not Allowed
lyoshenka commented 4 years ago

closed by #2768

mkroman commented 4 years ago

@lyoshenka that PR does not solve this particular issue - the PR just makes the lbrynet daemon able to run, but there's still a silent problem with connection resets from the API when using Python 3.8 as described in this issue which I haven't tracked down.

lyoshenka commented 4 years ago

@mkroman Thanks for letting me know. I've reopened the issue. If you do track down a solution, we would be grateful :-)

belikor commented 3 years ago

This error seems to be documented in the aiohttp project.

See https://github.com/aio-libs/aiohttp/issues/5818, https://github.com/aio-libs/aiohttp/issues/4724, https://github.com/aio-libs/aiohttp/issues/3507. In those cases they use Python 3.6 and 3.7, while the error in LBRY is only visible with Python 3.8+.

belikor commented 3 years ago

This issue should be solved with #3362.

The solution is just using a newer aiohttp. I tried aiohttp==3.6.2 because that is the version in the repositories in Ubuntu 20.04. To test I use Python 3.8 in Ubuntu, and Python 3.9 in Arch, and they both seem to work fine.