Closed wxl24life closed 10 months ago
sduzh
commented
1 year ago Hi @wxl24life, could you please follow these steps to help troubleshoot the issue:
show tablet 10644 to identify which table and partition the tablet 10644 belongs to.show partitions from <table>, and check the output to find the corresponding partition. Look at the visibleVersion and compactVersion values for that partition.sduzh
commented
1 year ago BTW，what object storage are you using?
wxl24life
commented
1 year ago BTW，what object storage are you using?
Aliyun OSS
wxl24life
commented
1 year ago Hi @wxl24life, could you please follow these steps to help troubleshoot the issue:
- Run
show tablet 10644to identify which table and partition the tablet 10644 belongs to.- Run
show partitions from <table>, and check the output to find the corresponding partition. Look at thevisibleVersionandcompactVersionvalues for that partition.
I have checked log, and found that the vacuum log has stopped at 6 am. The log started at around 18:00 yesterday. It continues for 12 hours. Not sure if there is any relation with OSS object storage policy?
sduzh
commented
1 year ago Hi @wxl24life, could you please follow these steps to help troubleshoot the issue:
- Run
show tablet 10644to identify which table and partition the tablet 10644 belongs to.- Run
show partitions from <table>, and check the output to find the corresponding partition. Look at thevisibleVersionandcompactVersionvalues for that partition.I have checked log, and found that the vacuum log has stopped at 6 am. The log started at around 18:00 yesterday. It continues for 12 hours. Not sure if there is any relation with OSS object storage policy?
Vacuum will stop for a partition if there have been no new imports or compactions within 12 hours (lake_autovacuum_stale_partition_threshold).
wxl24life
commented
1 year ago Hi @wxl24life, could you please follow these steps to help troubleshoot the issue:
- Run
show tablet 10644to identify which table and partition the tablet 10644 belongs to.- Run
show partitions from <table>, and check the output to find the corresponding partition. Look at thevisibleVersionandcompactVersionvalues for that partition.
visibleVersion and compactVersion have same value.
MySQL [(none)]> show tablet 10644
-> ;
+-----------+---------------------------+---------------------------+---------------------------+-------+---------+-------------+---------+--------+------------------------------------------------------------+
| DbName | TableName | PartitionName | IndexName | DbId | TableId | PartitionId | IndexId | IsSync | DetailCmd |
+-----------+---------------------------+---------------------------+---------------------------+-------+---------+-------------+---------+--------+------------------------------------------------------------+
| load_test | lineitem_csv_dk_replica_3 | lineitem_csv_dk_replica_3 | lineitem_csv_dk_replica_3 | 10171 | 10641 | 10640 | 10642 | true | SHOW PROC '/dbs/10171/10641/partitions/10640/10642/10644'; |
+-----------+---------------------------+---------------------------+---------------------------+-------+---------+-------------+---------+--------+------------------------------------------------------------+
1 row in set (0.002 sec)
MySQL [(none)]> use load_test;
Database changed
MySQL [load_test]> show partitions from lineitem_csv_dk_replica_3;
+-------------+---------------------------+----------------+----------------+-------------+--------+--------------+-------+-----------------+---------+----------+-----------+-----------------+------------+-------+-------+-------+
| PartitionId | PartitionName | CompactVersion | VisibleVersion | NextVersion | State | PartitionKey | Range | DistributionKey | Buckets | DataSize | RowCount | EnableDataCache | AsyncWrite | AvgCS | P50CS | MaxCS |
+-------------+---------------------------+----------------+----------------+-------------+--------+--------------+-------+-----------------+---------+----------+-----------+-----------------+------------+-------+-------+-------+
| 10640 | lineitem_csv_dk_replica_3 | 3 | 3 | 4 | NORMAL | | | l_orderkey | 96 | 22.8GB | 600037902 | true | false | 0.00 | 0.00 | 0.00 |
+-------------+---------------------------+----------------+----------------+-------------+--------+--------------+-------+-----------------+---------+----------+-----------+-----------------+------------+-------+-------+-------+
1 row in set (0.005 sec)In simple terms, if the last version of a partition was produced by a compaction job, vacuum will repeatedly try to delete files that have already been removed. The aliyun OSS will return OK when deleting a non-existent file, so it appears as though the same files are being deleted repeatedly, when in fact those files no longer exist.
sduzh
commented
1 year ago In simple terms, if the last version of a partition was produced by a compaction job, vacuum will repeatedly try to delete files that have already been removed. The aliyun OSS will return OK when deleting a non-existent file, so it appears as though the same files are being deleted repeatedly, when in fact those files no longer exist.
You can ignore these logs for now. Future versions will optimize this so that vacuum tasks that have already succeeded will not be repeatedly triggered.
github-actions[bot]
commented
10 months ago We have marked this issue as stale because it has been inactive for 6 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to StarRocks!
Steps to reproduce the behavior (Required)
In be.INFO log, the same data file is vacuumed periodically. It seems that the dat file has never been deleted at all.
Expected behavior (Required)
Real behavior (Required)
StarRocks version (Required)
runtime version build from main branch on 2023/07/19
select current_version()