Fix ES scroll bug - Githubissues

nestauk / dap_dv_backends

MIT License

0 stars 0 forks source link

Fix ES scroll bug #208

Closed doogyb closed 1 year ago

doogyb commented 1 year ago

We're overloading the ElasticSearch _scroll API when we want to migrate buckets back to an S3 bucket.

Specifically, we're getting a 429 Too Many Requests /_search/scroll response from the ES server.

Possible solutions:

Implement a cool down and try again method when we get the error
Use LogStash for ingesting/migrating data between ES and S3