search

toluaina / pgsync

Postgres to Elasticsearch/OpenSearch sync
https://pgsync.com
MIT License
1.1k stars 172 forks source link

getting 429 while sending bulk request #476

Open ChiragGangwani opened 11 months ago

ChiragGangwani commented 11 months ago

PGSync version: 2.5.1

Postgres version: 12

Elasticsearch version: latest

Redis version: latest

Python version: 3.10

Problem Description: While updating 9000 row in db some time it post to open search but some time it through an exception of bulk request.

we have small instance. these parameter are set

ELASTICSEARCH_CHUNK_SIZE=1000 ELASTICSEARCH_MAX_CHUNK_BYTES=3242880 ELASTICSEARCH_MAX_RETRIES=14 ELASTICSEARCH_QUEUE_SIZE=1 ELASTICSEARCH_STREAMING_BULK=True ELASTICSEARCH_THREAD_COUNT=1 ELASTICSEARCH_TIMEOUT=320 QUERY_CHUNK_SIZE=500

Error Message (if any): 

ERROR:pgsync.search_client: Exception TransportError(429, '429 Too Many Requests /_bulk')
data1-pgsync-1  | Traceback (most recent call last):
nsupegemini commented 8 months ago

Looks like ELASTICSEARCH_TIMEOUT is not used in bulk update https://github.com/toluaina/pgsync/blob/95116702c4b314d8b97696ef857cfe116241e236/pgsync/search_client.py#L188 @toluaina Could we please add timeout field to bulk update?