opensearch-project / opensearch-py

Python Client for OpenSearch
https://opensearch.org/docs/latest/clients/python/
Apache License 2.0
338 stars 170 forks source link

[FEATURE] Add support for yield to bulk inserts #504

Closed kxleee closed 10 months ago

kxleee commented 1 year ago

The bulk method cannot be used in combination with yield when batch inserts

dtaivpp commented 1 year ago

Hey @kxleee can you provide some additional information here to assist? For example are you using the async or sync client? Do you have some sample code you can share?

dblock commented 10 months ago

@kxleee Do you have an example please?

dblock commented 10 months ago

The helpers such as streaming_bulk do support yielding data, e.g. https://github.com/opensearch-project/opensearch-py/blob/main/samples/bulk/bulk-helpers.py#L84

def _generate_data() -> Any:
    for i in range(100):
        yield {"_index": index_name, "_id": i, "value": i}

succeeded = []
failed = []
for success, item in helpers.streaming_bulk(client, actions=_generate_data()):
    if success:
        succeeded.append(item)
    else:
        failed.append(item)

I am going to close this since we didn't hear from @kxleee with an example. Let's reopen if they mean something else.