search

apache / carbondata

High performance data store solution
carbondata.apache.org
Apache License 2.0
1.43k stars 704 forks source link

[CARBONDATA-4306] Fix Query Performance issue for Spark 3.1 #4252

Closed Indhumathi27 closed 2 years ago

Indhumathi27 commented 2 years ago

Why is this PR needed?

Some non-partition filters, which cannot be handled by carbon, is not pushed down to spark.

What changes were proposed in this PR?

If partition filters is non empty, then the filter column is not partition column, then push the filter to spark

Does this PR introduce any user interface change?

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4488/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6233/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/623/

kunal642 commented 2 years ago

retest this please

akashrn5 commented 2 years ago

LGTM

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6234/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4489/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/624/

kunal642 commented 2 years ago

LGTM