[CARBONDATA-4336] Table Status Versioning - Githubissues

apache / carbondata

High performance data store solution

carbondata.apache.org

Apache License 2.0

1.43k stars 704 forks source link

[CARBONDATA-4336] Table Status Versioning #4261

Closed Indhumathi27 closed 2 years ago

Indhumathi27 commented 2 years ago

Why is this PR needed?

Currently, carbondata will store the records of a transaction (load/insert/IUD/Add/drop segment) in a metadata file named `tablestatus’ which will be present in the Metadata directory. If the tablestatus file is lost, then the metadata for the transactions cannot be recovered directly, as there is no previous version file available for tablestatus. Hence, if we support versioning for tablestatus files, then it will be easy to recover the current version tablestatus meta from previous version tablestatus files.

Please refer Table Status Versioning & Recovery Tool for more info.

What changes were proposed in this PR?

-> On each transaction commit, committed the latest load metadata details to a new version file -> Updated the latest tablestatus version timestamp in the table properties [CarbonTable cache] and in the hive metastore -> Added a table status version tool which can recover the latest transaction details based on old version file

Does this PR introduce any user interface change?

Yes

Is any new testcase added?
Yes

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6279/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/669/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4535/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6280/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/670/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4536/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6281/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/671/

Indhumathi27 commented 2 years ago

retest this please

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/672/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6282/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4537/

AlphaAyush commented 2 years ago

retest this please

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4538/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6283/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/673/

AlphaAyush commented 2 years ago

retest this please

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4540/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6285/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6286/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/676/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6288/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4543/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/678/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4545/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6290/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/680/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6291/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/681/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4546/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4547/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6292/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6293/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/683/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4548/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4549/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6294/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/684/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4550/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6295/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/685/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6296/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4551/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/686/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6297/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4552/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/687/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6301/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4556/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/691/

Next