search

apache / carbondata

High performance data store solution
carbondata.apache.org
Apache License 2.0
1.43k stars 704 forks source link

[CARBONDATA-4347] Improve performance when delete empty partition directory #4288

Closed shenjiayu17 closed 9 months ago

shenjiayu17 commented 2 years ago

Why is this PR needed?

When delete multiple segments, carbondata tries to judge if partition directory is empty and delete it on each segments, but partiton path may be the same between two segments. So there is repetitive action.

What changes were proposed in this PR?

Collect a set of partition directories that are from segments to be deleted, judge empty and delete after all segments deleted.

Does this PR introduce any user interface change?

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4658/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6401/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/787/

shenjiayu17 commented 2 years ago

retest this please

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/788/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6402/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4659/

shenjiayu17 commented 2 years ago

retest this please

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/789/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6403/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4660/

shenjiayu17 commented 2 years ago

retest this please

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4662/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6405/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/791/

shenjiayu17 commented 2 years ago

retest this please

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4665/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6408/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/794/

xubo245 commented 1 year ago

Please fix CI error