search

apache / carbondata

High performance data store solution
carbondata.apache.org
Apache License 2.0
1.43k stars 704 forks source link

[CARBONDATA-4344] Create MV fails with "LOCAL_DICTIONARY_INCLUDE/LOCAL _DICTIONARY_EXCLUDE column: does not exist in table. Please check the DDL" error #4282

Closed ShreelekhyaG closed 2 years ago

ShreelekhyaG commented 2 years ago

Why is this PR needed?

Create MV fails with "LOCAL_DICTIONARY_INCLUDE/LOCAL _DICTIONARY_EXCLUDE column: does not exist in table. Please check the DDL" error. Error occurs only in this scenario: Create Table --> Load --> Alter Add Columns --> Drop table --> Refresh Table --> Create MV and not in direct scenario like: Create Table --> Load --> Alter Add Columns --> Create MV

What changes were proposed in this PR?

  1. After add column command, LOCAL_DICTIONARY_INCLUDE and LOCAL_DICTIONARY_EXCLUDE properties are added to the table even if the columns are empty. So, when MV is created next as LOCAL_DICTIONARY_EXCLUDE column is defined it tries to access its columns and fails. --> Added empty check before adding properties to the table to resolve this.

  2. In a direct scenario after add column, the schema gets updated in catalog table but the table properties are not updated. Made changes to update table properties to catalog table.

    Does this PR introduce any user interface change?

    • No

    Is any new testcase added?

    • Yes
CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4617/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6361/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/748/

ShreelekhyaG commented 2 years ago

retest this please

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4619/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6363/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/750/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.3/2/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4622/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6366/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/753/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6371/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4627/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/758/

Indhumathi27 commented 2 years ago

@ShreelekhyaG Please assign the JIRA with the PR description

Indhumathi27 commented 2 years ago

retest this please

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6372/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4628/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/759/

Indhumathi27 commented 2 years ago

LGTM