search

apache / carbondata

High performance data store solution
carbondata.apache.org
Apache License 2.0
1.43k stars 704 forks source link

[CARBONDATA-4285] Fix alter add complex columns with global sort compaction failure #4218

Closed maheshrajus closed 3 years ago

maheshrajus commented 3 years ago

Why is this PR needed?

Alter add complex columns with global sort compaction is failing due to 1) AOI exception : Currently creating default complex delimiter list in global sort compaction with size of 3. For map case need extra complex delimiter for handling the key-value 2) bad record handling: When we add complex columns after insert the data, complex columns has null data for previously loaded segments. this null value is going to treat as bad record and compaction is failed.

What changes were proposed in this PR?

1) In Global sort compaction flow create default complex delimiter with 4, as already doing in load flow. 2) Bad records handling pruned for compaction case. No need to check bad records for compaction as they are already checked while loading. previously loaded segments data we are inserting again in compaction case

Does this PR introduce any user interface change?

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/319/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5917/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4173/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/323/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5921/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4176/

CarbonDataQA2 commented 3 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4178/

CarbonDataQA2 commented 3 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5923/

CarbonDataQA2 commented 3 years ago

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/325/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4180/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5925/

CarbonDataQA2 commented 3 years ago

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/328/

CarbonDataQA2 commented 3 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5926/

CarbonDataQA2 commented 3 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4181/

Indhumathi27 commented 3 years ago

LGTM

ShreelekhyaG commented 3 years ago

LGTM

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/329/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4182/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5927/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5930/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4185/

CarbonDataQA2 commented 3 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/332/

CarbonDataQA2 commented 3 years ago

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/338/

CarbonDataQA2 commented 3 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4191/

CarbonDataQA2 commented 3 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/5937/

kunal642 commented 3 years ago

LGTM..PR needs some refactoring..please fix in another PR