[Enhancement]: Import supports overwrite mode

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications

https://milvus.io

Apache License 2.0

28.99k stars 2.79k forks source link

[Enhancement]: Import supports overwrite mode #34584

Closed bigsheeper closed 3 days ago

bigsheeper commented 1 month ago

Is there an existing issue for this?

[X] I have searched the existing issues

What would you like to be added?

Definition of import overwrite mode: Generate full L0 delete data for imported insert data to remove previously duplicated imported/inserted primary keys (similar to upsert).

Use case: Users need to import data in bulk daily, and the imported data is prone to duplication.

Prerequisite: Shardleader supports forwarding incremental L0 segments. After load refresh done, all imported L0 deletes are visible in all querynodes.

Why is this needed?

No response

Anything else?

No response

bigsheeper commented 1 month ago

/priority: medium

sre-ci-robot commented 1 month ago

@bigsheeper: The label(s) priority/medium cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/milvus-io/milvus/issues/34584#issuecomment-2220309408): >/priority: medium Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

stale[bot] commented 1 week ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.