Closed ryandotsmith closed 7 months ago
It is common for free node services to aggressively rate limit requests. This impacts both backfill and incremental indexing.
For backfilling, it is advisable to use low values for batch_size and concurrency. For example: batch_size=10 and concurrency=1.
For incremental indexing, it is now advisable to increase the poll_duration for a source. For example: poll_duration=1s
Both of these knobs will reduce the amount of requests that are sent to a hosted node.
It is common for free node services to aggressively rate limit requests. This impacts both backfill and incremental indexing.
For backfilling, it is advisable to use low values for batch_size and concurrency. For example: batch_size=10 and concurrency=1.
For incremental indexing, it is now advisable to increase the poll_duration for a source. For example: poll_duration=1s
Both of these knobs will reduce the amount of requests that are sent to a hosted node.