Closed wangting0128 closed 3 months ago
/unassign
/assign @chyezh could you help on verifying this?
From log
2024-05-19 18:50:51.497 | (no unique labels) | [2024/05/19 18:50:51.497 +00:00] [INFO] [datacoord/garbage_collector.go:517] ["garbageCollector will recycle index files"] [buildID=449193006024386963]
the index file has been gc, but minio didn't delete it.
I have tried to delete it at offline, seems that delete method perform correctly at client-side.
204 no content
is returned from server side.
It's a success http code at minio client. https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html
it's a bug of minio?
it can not be deleted on minio console too.
it can not be deleted on minio console too.
Another problem with this issue is that the gc time seems to be inconsistent with the configured time.
DropCollection
don't update DroppedAt
field of segment.
so no tolerance time wait when gc segment.
will fix it in next version.
2024-05-16 06:33:15.336(no unique labels)[2024/05/16 06:33:15.336 +00:00] [INFO] [datacoord/services.go:554] ["receive DropVirtualChannel request"] [traceID=3e9b4e63e2e7e2339060c74aa6c2de90] [channelName=by-dev-rootcoord-dml_0_449193005785809078v0]2024-05-16 06:33:15.336(no unique labels)[2024/05/16 06:33:15.336 +00:00] [DEBUG] [datacoord/meta.go:842] ["meta update: update drop channel segment info"] [channel=by-dev-rootcoord-dml_0_449193005785809078v0] | 2024-05-16 06:33:15.336 | (no unique labels) | [2024/05/16 06:33:15.336 +00:00] [INFO] [datacoord/services.go:554] ["receive DropVirtualChannel request"] [traceID=3e9b4e63e2e7e2339060c74aa6c2de90] [channelName=by-dev-rootcoord-dml_0_449193005785809078v0] | | | | 2024-05-16 06:33:15.336 | (no unique labels) | [2024/05/16 06:33:15.336 +00:00] [DEBUG] [datacoord/meta.go:842] ["meta update: update drop channel segment info"] [channel=by-dev-rootcoord-dml_0_449193005785809078v0] |
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
2024-05-16 06:33:15.336 | (no unique labels) | [2024/05/16 06:33:15.336 +00:00] [INFO] [datacoord/services.go:554] ["receive DropVirtualChannel request"] [traceID=3e9b4e63e2e7e2339060c74aa6c2de90] [channelName=by-dev-rootcoord-dml_0_449193005785809078v0] |
| | 2024-05-16 06:33:15.336 | (no unique labels) | [2024/05/16 06:33:15.336 +00:00] [DEBUG] [datacoord/meta.go:842] ["meta update: update drop channel segment info"] [channel=by-dev-rootcoord-dml_0_449193005785809078v0] |
2024-05-16 06:33:15.336(no unique labels)[2024/05/16 06:33:15.336 +00:00] [DEBUG] [datacoord/meta.go:1587] ["updating segment state and updating metrics"] [segmentID=449193005786090852] ["old state"=Flushed] ["new state"=Dropped] ["# of rows"=350000] | | | 2024-05-16 06:33:15.336 | (no unique labels) | [2024/05/16 06:33:15.336 +00:00] [DEBUG] [datacoord/meta.go:1587] ["updating segment state and updating metrics"] [segmentID=449193005786090852] ["old state"=Flushed] ["new state"=Dropped] ["# of rows"=350000] |
| | 2024-05-16 06:33:15.336 | (no unique labels) | [2024/05/16 06:33:15.336 +00:00] [DEBUG] [datacoord/meta.go:1587] ["updating segment state and updating metrics"] [segmentID=449193005786090852] ["old state"=Flushed] ["new state"=Dropped] ["# of rows"=350000] |
2024-05-16 06:33:15.336(no unique labels)[2024/05/16 06:33:15.336 +00:00] [DEBUG] [datacoord/meta.go:1587] ["updating segment state and updating metrics"] [segmentID=449193005786161686] ["old state"=Flushed] ["new state"=Dropped] ["# of rows"=300000] | | | 2024-05-16 06:33:15.336 | (no unique labels) | [2024/05/16 06:33:15.336 +00:00] [DEBUG] [datacoord/meta.go:1587] ["updating segment state and updating metrics"] [segmentID=449193005786161686] ["old state"=Flushed] ["new state"=Dropped] ["# of rows"=300000]
| | 2024-05-16 06:33:15.336 | (no unique labels) | [2024/05/16 06:33:15.336 +00:00] [DEBUG] [datacoord/meta.go:1587] ["updating segment state and updating metrics"] [segmentID=449193005786161686] ["old state"=Flushed] ["new state"=Dropped] ["# of rows"=300000]
it can not be deleted on minio console too.
Another problem with this issue is that the gc time seems to be inconsistent with the configured time.
@wangting0128 ,should be fixed, please verify these problem.
image:2.4-20240614-fd1c7b1a-amd64
No data is left on minio
Is there an existing issue for this?
Environment
Current Behavior
argo task: fouramf-sbfcd
server:
minio garbage collection after 40 minutes
residual data
client pod name: fouramf-sbfcd-833673770 client log:
Expected Behavior
No response
Steps To Reproduce
Milvus Log
minio data:
Anything else?
milvus.yaml