search

apache / carbondata

High performance data store solution
carbondata.apache.org
Apache License 2.0
1.43k stars 704 forks source link

[CARBONDATA-4300] Clean files command supports specify segment ids #4232

Open jack86596 opened 3 years ago

jack86596 commented 3 years ago

Why is this PR needed?

Currently clean files command will delete all the Marked for Delete and Compacted segments after the number of theses segments reaches carbon.invisible.segments.preserve.count, this delete operation may take lots of time and user cannot decide to only delete some of these segments. It is better to enhance clean files command to allow specify the segments to be deleted.

What changes were proposed in this PR?

  1. Clean files command supports specify segment ids, syntax is "clean files for table table_name options("segment_ids"="id1,id2,id3...")". If specified segment ids, then only the segment with these ids will be delete physically.

  2. Refactoring lock taken: during clean files, take the tablestatus lock at the begining and release the lock at the end, and during lock taken period, only read tablestatus file one time(before there could be 10+) and all operations are done on it like change the visibility of segment, move visibility = false segment to tablestatus.history file.

    Does this PR introduce any user interface change?

    • Yes. One more option is added for clean files command: segment_ids. Value is the segment ids user wants to delete. Only Marked for Delete and Compacted segment ids are valid. If invalid ids are given, operation will fail directly. If segments are specified, force option will be ignored. CLEAN FILES FOR TABLE TABLE_NAME options('segment_ids'='0,1,2')

    Is any new testcase added?

    • Yes
CarbonDataQA2 commented 3 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6042/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/432/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4298/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6043/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4299/

CarbonDataQA2 commented 3 years ago

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/433/

jack86596 commented 3 years ago

retest this please

CarbonDataQA2 commented 3 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6044/

CarbonDataQA2 commented 3 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4300/

CarbonDataQA2 commented 3 years ago

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/434/

akashrn5 commented 3 years ago

@jack86596 this is behavioral and functional change. So instead of directly raising PR with more code changes, better to first raise discussion in community and take the inputs and then do the changes accordingly.

jack86596 commented 3 years ago

OK, i will raise a discussion in the mail list.