search

baking-bad / tzkt

😼 Awesome Tezos blockchain indexer and API
https://tzkt.io
MIT License
183 stars 35 forks source link

1.6.0: Timeout and the requested operation requires an element of type 'Array', but the target element has type 'Object'. #58

Closed matyushkins closed 3 years ago

matyushkins commented 3 years ago

Hi. Sync 1.6.0

info: Tzkt.Sync.Program[0]
      Initialize database
info: Tzkt.Sync.Program[0]
      Database initialized
warn: Tzkt.Sync.Services.Observer[0]
      Observer started
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app
info: Tzkt.Sync.Services.Observer[0]
      State initialized: [1589246:BL2DKZ6CZfzG9wMjVCttdt5wK8fRydDvHXyVDnt3XJzEr2uiG5r]
info: Tzkt.Sync.Services.QuotesService[0]
      Quote provider: TzktQuotesProvider (Async)
info: Tzkt.Sync.Services.Observer[0]
      Quotes initialized: [1589246]
info: Tzkt.Sync.Services.Observer[0]
      Synchronization started
fail: Tzkt.Sync.Services.Observer[0]
      Failed to check updates. The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
fail: Tzkt.Sync.Services.Observer[0]
      Failed to check updates. The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. The requested operation requires an element of type 'Array', but the target element has type 'Object'.
fail: Tzkt.Sync.Services.Observer[0]
      Failed to apply updates. The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
fail: Tzkt.Sync.Services.Observer[0]
Groxan commented 3 years ago

Hey! That's related to yesterday's Granada upgrade that happened at the block 1,589,247. The issue is that the Tezos node is no longer able to normally process /helpers/endorsing_rights?cycle={n} RPC request, which is used by the indexer during the protocol activation. I guess this is because the number of endorsements per cycle was increased (multiplied by 16) in Granada and it seems too much for the node to process in a single request.

Yesterday we released a patch that replaces the /endorsing_rights?cycle= RPC request with 8192 /endorsing_rights?level= RPC requests, so the node shouldn't fail with timeout anymore. Yes, it's very inefficient and slow, but we don't see another way how to solve the issue yet.

Also, we uploaded a fresh snapshot which is already on Granada.

So, you have three options:

  1. Increase the TezosNode.Timeout to 1800 sec or more in the appsettings or via env vars and hope that your node will manage to process the /endorsing_rights?cycle= RPC request. It will be 5 requests (for 5 future cycles), so be ready to wait about an hour.
  2. Update the indexer (only the tzkt-sync binaries) to the latest commit (or docker image with the tag latest) and run. The Granada activation will take about 30-50min, because 8192blocks*5cycles RPC requests will be sent.
  3. Update the indexer and restore the DB from the latest snapshot where the Granada activation was already passed.

Feel free to reopen the issue if you face any problems.

matyushkins commented 3 years ago

Ok. As we noticed, the RPC node takes a long time to respond even to simple requests. Most likely this is the problem with your requests.