Open lentitude2tk opened 2 months ago
/assign
I'd close this issue for a zilliz cloud issue.
I'd close this issue for a zilliz cloud issue.
Why was this issue closed? This problem was discovered in the cloud, but the root cause is a series of issues caused by the kernel's inability to strictly enforce disk quota limits at the database level.
seems that there is no quick fix for that?
if insertion is quota limited, then the data size is already more than a same memory cluster can load.
Unless for all backup request we remove the check
if insertion is quota limited, then the data size is already more than a same memory cluster can load.
Yes, as long as the user encounters this quota limited message, it means that the current data size has already exceeded the normal quota, which will affect subsequent processes.
Unless for all backup request we remove the check
In fact, the current operation is similar. When encountering such errors, especially during resume, we can only lift the restriction to allow the process to continue, but currently, this is done manually.
A simple solution currently considered is to expose a parameter in the bulkInsert Option that allows skipping validation. One benefit is that different restrictions can be applied for stop/resume and migrate. For normal user stop/resume operations, it should be ensured that data can be written normally and that the user can resume normally after stopping. However, during migrateFrom, restrictions should be applied to prevent users from writing a large amount of data exceeding the diskQuota into Milvus.
if insertion is quota limited, then the data size is already more than a same memory cluster can load.
Yes, as long as the user encounters this quota limited message, it means that the current data size has already exceeded the normal quota, which will affect subsequent processes.
Unless for all backup request we remove the check
In fact, the current operation is similar. When encountering such errors, especially during resume, we can only lift the restriction to allow the process to continue, but currently, this is done manually.
A simple solution currently considered is to expose a parameter in the bulkInsert Option that allows skipping validation. One benefit is that different restrictions can be applied for stop/resume and migrate. For normal user stop/resume operations, it should be ensured that data can be written normally and that the user can resume normally after stopping. However, during migrateFrom, restrictions should be applied to prevent users from writing a large amount of data exceeding the diskQuota into Milvus.
Agree we can add a config to ignore the disk size check. comment? @bigsheeper
/open
Agree we can add a config to ignore the disk size check. comment? @bigsheeper
Yep, I also think so.
we should still enable disk quota limit
We can add an option to the import request to skip the disk quota check, which would give us more flexibility. And we don't need to worry about cloud users bypassing the check by setting this option, as they don't have access to the bulkinsert interface.
@lentitude2tk pr merged, this enhancement will be released at vertion 2.4.7
/assign @lentitude2tk /unassign
Is there an existing issue for this?
Environment
Current Behavior
Expected Behavior
If the user's data can be written normally, the backup and subsequent recovery should also work properly. Here are some potential solutions for reference, with the specifics open to kernel discussion:
Steps To Reproduce
Milvus Log
https://grafana.op.zillizcloud.com/explore?orgId=1&left=%7B%22datasource%22:%22Loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bcluster%3D%5C%22milvus-gcp-us-west1-1%5C%22,namespace%3D%5C%22milvus-in01-631a262f0093e3a%5C%22,pod%3D~%5C%22in01-631a262f0093e3a-milvus-.*%5C%22%7D%7C%3D%5C%22450140497833660850%5C%22%22%7D%5D,%22range%22:%7B%22from%22:%22now-1h%22,%22to%22:%22now%22%7D%7D
Anything else?
No response