lbryio / lbry-sdk

The LBRY SDK for building decentralized, censorship resistant, monetized, digital content apps.
https://lbry.com
MIT License
7.19k stars 482 forks source link

error on large resolve / sync calls - response too large (over 1,000,000 bytes') #1999

Closed tzarebczan closed 5 years ago

tzarebczan commented 5 years ago
## The Issue Have lots of subscriptions in my app and ran into this error when running the below. the size should be no different than resolving 100 claims, which we do for homepage content. ``` "result": { "error": "(-32600, 'response too large (over 1,000,000 bytes')" ``` ``` {"jsonrpc":"2.0","method":"resolve","params":{"urls":["lbry://@GaminGHD#086d2bacf441cef45ff15b5afe163d0b03a9f7ea","lbry://@oniblackmage#0ddd797e39c5fe706ed4ee62f96becb792aee03a","lbry://@CrazyFoxMovies#1516361918bfd02ddd460489f438e153c918521c","lbry://@TechFox#2305db36455aa0d18571015b9e9bd0950262aa0f","lbry://@TomTestNewChan#27a5f8e013ab04b0ac11c736bb8f7c91e115ba61","lbry://@FreeKeene#27aad3c089dac2dbd205492d2cab32b7981f22d8","lbry://@Dota2Divine#2d2c1a3dc98ca868d1f2bfe76b8a1bb820857c5a","lbry://@Karolajn#319a7ffcebe1c4d23b5b5fc2e66452695e030e88","lbry://@bluedrake42#32de523ba228dd3f3159eb5a6cc07b6fd51f4dc0","lbry://@grabthegames#379290e593323eb208070269ea2c29a4ddf64985","lbry://@OYENEWS#3cabf5f5f01c5bc244c96ab20f709b6a7668af5f","lbry://@Gunslinger-Media#48f966962b51459f00c55f76eb62960e9705fa79","lbry://@barnacules#4e17d248adc0128afe969c2e1327e10afd9cb921","lbry://@miss-zi-zi#4ecc029cac207edb5bb7770947ebb9e84f7ebab9","lbry://@NaomiBrockwell#4ee7cfaf1fc50a6df858ed0b99c278d633bccca9","lbry://@MinutePhysics#589276465a23c589801d874f484cc39f307d7ec7","lbry://@brightsunfilms#5a1b164d0a2e7adf1db08d7363ea1cb06c30cd74","lbry://@nickatnyte#5b7c7a202201033d99e1be2930d290c127c0f4fe","lbry://@Itsrucka#6184648aab0431c4c95c649072d1f9ff08b9bb7c","lbry://@MothersBasement#65dee913b4f26144d573118b2e87704e03d01b8f","lbry://@ILikeToMakeStuff#67575d8ec8f03ccde78cbd61e650f0418cf9cbed","lbry://@internalch#8ed685db6135e5e1ecc1df56edb2e70e818b3779","lbry://@CryptoCandor#9152f3b054f692076a6882d1b58a30e8781cc8e6","lbry://@Acituanbus#aeb882df790ac1c25697954b2bf5cbb3a4ed06f9","lbry://@3dprinting#b0cb60d0d53176717f3ed8df2b076aec611d8193","lbry://@Porn#b5e2fb3e2f8ab54cff026924e263ca816426a09b","lbry://@3Blue1Brown#ba79c80788a9e1751e49ad401f5692d86f73a2db","lbry://@thomaszarebczan#bc40c0f1039ba79341df42ff204dcf0908198694","lbry://@JordanBPeterson#c5724e280283cd985186af9a62494aae377daabd","lbry://@ShutupAndPlay#d4f071db22ee06501924d4c77d73e67c4f5cbe80","lbry://@jiggytom#e48d2b50501159034f68d53321f67b8aa5b1d771","lbry://@TheCryptoLark#e50beec197c7d1cd2f9e6ce0f4176fee9d4949b9","lbry://@Lunduke#e8f68563d242f6ac9784dcbc41dd86c28a9391d6","lbry://@Crypt0#f0cb90bbe9ba6cf958a3fc2261a6743997e13551","lbry://@TipWhatYouLike#f5ca96fa22dc3f14445ba001da8df7ec897f8521","lbry://@import-test#f6034d82c1e238ef876dd28dedc4afb79b1fd87d","lbry://@TheLinuxGamer#feb61536c007cdf4faeeaab4876cb397feaf6b51"]},"id":1553619477257} ``` ## System Configuration

Anything Else

Screenshots

Internal Use

Acceptance Criteria

1. 2. 3.

Definition of Done

tzarebczan commented 5 years ago

Also running into this when syncing the spee.ch wallet locally to my spv server. On lbryumx1/2, get a different error (not sure if related).

Task exception was never retrieved
future: <Task finished coro=<BaseLedger.update_history() done, defined at c:\users\thoma\documents\lbry\fresh3-27\lib\site-packages\torba\client\baseledger.py:369> exception=RPCError(-32600, 'response too large (over 1,000,000 bytes')>
Traceback (most recent call last):
  File "c:\users\thoma\documents\lbry\fresh3-27\lib\site-packages\torba\client\baseledger.py", line 379, in update_history
    remote_history = await self.network.get_history(address)
  File "c:\users\thoma\documents\lbry\fresh3-27\lib\site-packages\torba\client\basenetwork.py", line 32, in send_request
    raise e
  File "c:\users\thoma\documents\lbry\fresh3-27\lib\site-packages\torba\client\basenetwork.py", line 29, in send_request
    return await super().send_request(method, args)
  File "c:\users\thoma\documents\lbry\fresh3-27\lib\site-packages\torba\rpc\session.py", line 481, in send_request
    raise result
torba.rpc.jsonrpc.RPCError: (-32600, 'response too large (over 1,000,000 bytes')

x1/x2


future: <Task finished coro=<BaseLedger.subscribe_account() done, defined at c:\users\thoma\documents\lbry\fresh3-27\lib\site-packages\torba\client\baseledger.py:344> exception=InterfaceError('Error binding parameter 0 - probably unsupported type.')>
Traceback (most recent call last):
  File "c:\users\thoma\documents\lbry\fresh3-27\lib\site-packages\torba\client\baseledger.py", line 347, in subscribe_account
    await account.ensure_address_gap()
  File "c:\users\thoma\documents\lbry\fresh3-27\lib\site-packages\torba\client\baseaccount.py", line 375, in ensure_address_gap
    new_addresses = await address_manager.ensure_address_gap()
  File "c:\users\thoma\documents\lbry\fresh3-27\lib\site-packages\torba\client\baseaccount.py", line 139, in ensure_address_gap
    new_keys = await self._generate_keys(start, end-1)
  File "c:\users\thoma\documents\lbry\fresh3-27\lib\site-packages\torba\client\baseaccount.py", line 147, in _generate_keys
    await self.account.ledger.db.add_keys(self.account, self.chain_number, keys)
  File "c:\users\thoma\documents\lbry\fresh3-27\lib\site-packages\torba\client\basedatabase.py", line 511, in add_keys
    for position, pubkey in keys)
  File "C:\Users\thoma\AppData\Local\Programs\Python\Python37-32\lib\concurrent\futures\thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "c:\users\thoma\documents\lbry\fresh3-27\lib\site-packages\torba\client\basedatabase.py", line 61, in __run_transaction
    result = fun(self.connection, *args, **kwargs)  # type: ignore
  File "c:\users\thoma\documents\lbry\fresh3-27\lib\site-packages\torba\client\basedatabase.py", line 39, in __executemany_in_a_transaction
    return conn.executemany(*args, **kwargs)
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.```
tzarebczan commented 5 years ago

This max_size parameter can be adjusted on the wallet server. @shyba can we do this with the next restarts. Will see if 1.5 MB fixes the spee.ch wallet sync issue, but I'm more concerned about larger resolve calls (i.e. homepage, list of channels).

tzarebczan commented 5 years ago

This has been adjusted on the wallet server side, closing.