Open shepilov opened 1 year ago
Isn’t it simpler to add a delay of 2 seconds after the bulk operation ? When the data is small like in e2e tests 2s is way more than enough to make sure the data is indexed and ready for retrievalOn 1 Jun 2023, at 22:10, Anton Shepilov @.***> wrote: One of the ways to do it is explicitly make a call to ElasticSearch. Before that we need to add a pipeline to Elastic, to set this date after the doc was indexed https://www.elastic.co/guide/en/elasticsearch/reference/7.10/accessing-data-in-pipelines.html#accessing-ingest-metadata And then you can check your doc by id Another way is to add a listener to the Elastic bulk operations and set timeout according to https://www.elastic.co/guide/en/elasticsearch/reference/current/near-real-time.html
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>
In Apache James we flush the index after indexing CF https://github.com/apache/james-project/blob/a0fbd09941ac18ec6c22d739ecbcce8d83fea6d6/backends-common/opensearch/src/test/java/org/apache/james/backends/opensearch/DockerOpenSearch.java#L332
BTW we NEED to align the search engine on OpenSearch.
Isn’t it simpler to add a delay of 2 seconds after the bulk operation ? When the data is small like in e2e tests 2s is way more than enough to make sure the data is indexed and ready for retrieval @RomaricMourgues Yes, I've maid right now like this, but it's not stable, one of 10 runs fails
In Apache James we flush the index after indexing CF https://github.com/apache/james-project/blob/a0fbd09941ac18ec6c22d739ecbcce8d83fea6d6/backends-common/opensearch/src/test/java/org/apache/james/backends/opensearch/DockerOpenSearch.java#L332
BTW we NEED to align the search engine on OpenSearch.
thanks @chibenwa do you have the helm script for OpenSearch? and is the flush API method synchronous?
thanks @chibenwa do you have the helm script for OpenSearch?
Ops chose to deploy DBs outside of K8S using old-fashionned ansible deployments. Persistance volumes is somewhat of a complex topic... And of course we have an ansible deployment available.
Cc @ducnm0711
and is the flush API method synchronous?
Yes
we can spinup OpenSearch helm chart in tdrive-dev cluster if you just need to test the API for response. OpenSearch v2.7.0 is ok or we have other specific version requirement?
we can spinup OpenSearch helm chart in tdrive-dev cluster if you just need to test the API for response. OpenSearch v2.7.0 is ok or we have other specific version requirement?
yes, it should be fine
One of the ways to do it is explicitly make a call to ElasticSearch. Before that we need to add a pipeline to Elastic, to set this date after the doc was indexed https://www.elastic.co/guide/en/elasticsearch/reference/7.10/accessing-data-in-pipelines.html#accessing-ingest-metadata And then you can check your doc by
id
Another way is to add a listener to the Elastic bulk operations and set timeout according to https://www.elastic.co/guide/en/elasticsearch/reference/current/near-real-time.html