The syncing will should try an infinite number of times retying with a page size of 1 many times over until an expected response is received. Which should helpfully get around any temporary network interruptions.
This PR also includes several logging improvements that helped to debug this issue and get stats on query, parsing, and tx processing times.
Page size back-off details
eg: using endpoint ${RPC_API}/tx_search?query="${query}"&per_page=${perPage}&page=${page} demonstrating the paging mechanism when some errors are met.
request 1:
perPage=100
page=1
completed: 100 items
request 2:
perPage=100
page=2
completed: 200 items
request 3:
perPage=100
page=3
ERROR & retry 1:
delay: 1s
perPage=10
page=21
ERROR & retry 2:
delay: 2s
perPage=1
page=201
completed: 201 items
request 4:
perPage=1
page=202
completed: 202 items
...
request 12:
perPage=1
page=210
completed: 210 items
request 13:
perPage=10
page=22
completed: 220 items
request 14:
perPage=10
page=23
completed: 230 items
...
request 21:
perPage=10
page=30
completed: 300 items
request 22:
perPage=100
page=4
completed: 400 items
Some logs showing the syncing process with a network interruption for at least 10 minutes between setup and October 20, 2023 at 12:44 (UTC+10:00).
October 20, 2023 at 12:50 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=301
October 20, 2023 at 12:49 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=301
October 20, 2023 at 12:49 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=301
October 20, 2023 at 12:49 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=301
October 20, 2023 at 12:48 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=10&page=31
October 20, 2023 at 12:48 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=100&page=4
October 20, 2023 at 12:48 (UTC+10:00) info: transaction import progress: 10.2% (300 items) (~829ms per item)
October 20, 2023 at 12:47 (UTC+10:00) info: transaction import progress: 7.9% (200 items) (~564ms per item)
October 20, 2023 at 12:46 (UTC+10:00) info: transaction import progress: 4.0% (100 items) (~1003ms per item)
October 20, 2023 at 12:45 (UTC+10:00) info: transaction import progress: 3.7% (90 items) (~776ms per item)
October 20, 2023 at 12:45 (UTC+10:00) info: transaction import progress: 3.3% (80 items) (~853ms per item)
October 20, 2023 at 12:45 (UTC+10:00) info: transaction import progress: 3.0% (70 items) (~669ms per item)
October 20, 2023 at 12:45 (UTC+10:00) info: transaction import progress: 2.6% (60 items) (~1019ms per item)
October 20, 2023 at 12:45 (UTC+10:00) info: transaction import progress: 2.2% (50 items) (~905ms per item)
October 20, 2023 at 12:45 (UTC+10:00) info: transaction import progress: 1.8% (40 items) (~1109ms per item)
October 20, 2023 at 12:45 (UTC+10:00) info: transaction import progress: 1.5% (30 items) (~977ms per item)
October 20, 2023 at 12:44 (UTC+10:00) info: transaction import progress: 1.0% (20 items) (~688ms per item)
October 20, 2023 at 12:44 (UTC+10:00) info: transaction import progress: 0.5% (10 items) (~2684ms per item)
October 20, 2023 at 12:44 (UTC+10:00) info: transaction import progress: 0.4% (9 items) (~2417ms per item)
October 20, 2023 at 12:44 (UTC+10:00) info: transaction import progress: 0.4% (8 items) (~3334ms per item)
October 20, 2023 at 12:44 (UTC+10:00) info: transaction import progress: 0.4% (7 items) (~1084ms per item)
October 20, 2023 at 12:44 (UTC+10:00) info: transaction import progress: 0.3% (6 items) (~2537ms per item)
October 20, 2023 at 12:44 (UTC+10:00) info: transaction import progress: 0.3% (5 items) (~1744ms per item)
October 20, 2023 at 12:44 (UTC+10:00) info: transaction import progress: 0.2% (4 items) (~2013ms per item)
October 20, 2023 at 12:44 (UTC+10:00) info: transaction import progress: 0.2% (3 items) (~2060ms per item)
October 20, 2023 at 12:44 (UTC+10:00) info: transaction import progress: 0.1% (2 items) (~2235ms per item)
October 20, 2023 at 12:44 (UTC+10:00) info: transaction import progress: 0.1% (1 items) (~765504ms per item)
October 20, 2023 at 12:44 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:44 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:43 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:43 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:43 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:42 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:42 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:42 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:42 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:41 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:41 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:41 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:41 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:41 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:40 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:40 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:40 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:39 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:37 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=1&page=1
October 20, 2023 at 12:35 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=10&page=1
October 20, 2023 at 12:33 (UTC+10:00) error: Could not fetch txs from URL: http://follower-node:26657/tx_search?query="tx.height%3E%3D0%20AND%20message.module%3D'dex'"&per_page=100&page=1
October 20, 2023 at 12:31 (UTC+10:00) info: transaction import starting
October 20, 2023 at 12:31 (UTC+10:00) info: Server running on http://0.0.0.0:8000
October 20, 2023 at 12:31 (UTC+10:00) info: connected to API: http://follower-node:26657
October 20, 2023 at 12:31 (UTC+10:00) info: testing connection to API: http://follower-node:26657
October 20, 2023 at 12:31 (UTC+10:00) dist/server.js 96.2kb
October 20, 2023 at 12:31 (UTC+10:00) ⚡ Done in 174ms
October 20, 2023 at 12:31 (UTC+10:00) > duality-hapi-indexer@0.3.0 esbuild
October 20, 2023 at 12:31 (UTC+10:00) > esbuild src/server.ts --bundle --platform=node --packages=external --loader:.sql=text --outdir=dist
October 20, 2023 at 12:31 (UTC+10:00) > duality-hapi-indexer@0.3.0 start
October 20, 2023 at 12:31 (UTC+10:00) > npm run esbuild && node dist/server.js
This PRs allows fetching from RPC txs endpoint to be retried and broken down into smaller page sizes if needed.
The below paging mechanism can be mostly removed when targeting a CosmosSDK v0.47.4+ chain by querying txs from the REST API instead of the RPC API: see branch https://github.com/duality-labs/hapi-indexer/pull/new/fix-syncing-with-large-txs-pages.
The syncing will should try an infinite number of times retying with a page size of 1 many times over until an expected response is received. Which should helpfully get around any temporary network interruptions.
This PR also includes several logging improvements that helped to debug this issue and get stats on query, parsing, and tx processing times.
Page size back-off details
eg: using endpoint
${RPC_API}/tx_search?query="${query}"&per_page=${perPage}&page=${page}demonstrating the paging mechanism when some errors are met.perPage=100page=1perPage=100page=2perPage=100page=3ERROR& retry 1:perPage=10page=21ERROR& retry 2:perPage=1page=201perPage=1page=202perPage=1page=210perPage=10page=22perPage=10page=23perPage=10page=30perPage=100page=4Some logs showing the syncing process with a network interruption for at least 10 minutes between setup and October 20, 2023 at 12:44 (UTC+10:00).