opensearch-project / opensearch-net

OpenSearch .NET Client
Apache License 2.0
102 stars 49 forks source link

Change Bulk All to support Update requests #690

Closed aneetas closed 2 months ago

aneetas commented 3 months ago

Is your feature request related to a problem?

We are looking to use Update Requests so we can do a partial update of an Open Search document (using DocAsUpsert). We need to do a bulk transaction with these update requests. The _openSearchClient.Bulk(bulkRequest) currently supports this, however this method does not provide support for batching & retries. The _openSearchClient.BulkAll(documents,...) does support batching & retries, however this doesn't support UpdateRequests with the DocAsUpsert property.

What solution would you like?

We would like the _openSearchClient.BulkAll(documents,...) method to be updated to support BulkOperations such as Update, Delete. The BulkAll method currently only takes in documents and indexes them.

What alternatives have you considered?

A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?

Add any other context or screenshots about the feature request here.

dblock commented 3 months ago

Makes sense. Assuming this is supported by OpenSearch server, want to try to contribute the feature?

Xtansia commented 3 months ago

@aneetas You can customise how the batches are converted into actual bulk requests using .BufferToBulk, assuming all the items should be update/upsert operations:

client.BulkAll(
            documents, 
            bulkAll => bulkAll
                .BufferToBulk((bulk, batch) => bulk
                    .UpdateMany(batch, (update, doc) => update
                        .Doc(doc)
                        .DocAsUpsert()
                    )
                )
            );

Do note however, BulkAll parallelises requests by default, so updates may happen out of order.

aneetas commented 2 months ago

@Xtansia thank you! I think that's all I need for the moment.