search

apache / carbondata

High performance data store solution
carbondata.apache.org
Apache License 2.0
1.43k stars 704 forks source link

[CARBONDATA-4346] Remove list files while query and invalid cache #4287

Open ShreelekhyaG opened 2 years ago

ShreelekhyaG commented 2 years ago

Why is this PR needed?

  1. Performance degradation for Incremental updates is observed in the partition table.

    • During the update, in the prune step we are listing files from segment path to get the carbondata files and create fileNameToMetaInfoMappingmap. On incremental update for partition table, the number of invalid files keep on increasing each time which is causing the degradation in createCarbonDataFileBlockMetaInfoMappingmethod.
    • Example: Assume a single partition with 1000 carbondata files. Perform 1st update: adds 900 new carbondata files. Perform 2nd update (same update query): adds another 900 carbondata files. Now the files added by 1st update are invalid. Perform query: It does list files. Here, considers invalid files also and adds to fileNameToMetaInfoMappingmap. The number of invalid files keeps on increasing with each update which is causing the degradation in creating fileNameToMetaInfoMappingmap.

  2. Invalid segments cache is not removed after delete/update.

    What changes were proposed in this PR?

  3. Instead of listing files, made a change to get the carbon file from the file name and create BlockMetaInfo directly in createBlockMetaInfo. Impact when tested on a single partition with 100 segments:

    • There is significant improvement observed in the Incremental update operation.
    • 95% improvement seen in 1st time select count(*) operation. Because in select count(*) flow it was listing files for each segment and the map was not reused. Impact when tested on a non-partition table with 100 segments:
    • Almost the same or no improvement for the non-partition table

  4. Clearing invalid/deleted segments from cache after delete and update.

    Does this PR introduce any user interface change?

    • No

    Is any new testcase added?

    • Yes
CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/772/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6386/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4641/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6387/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4642/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/773/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6390/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4645/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/776/

Indhumathi27 commented 2 years ago

LGTM

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6393/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4650/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/779/

ShreelekhyaG commented 2 years ago

retest this please

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6395/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4652/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/781/

ShreelekhyaG commented 2 years ago

retest this please

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4653/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/782/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6396/

akashrn5 commented 2 years ago

LGTM

ShreelekhyaG commented 2 years ago

retest this please

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12602/job/ApacheCarbonPRBuilder2.3/6397/

CarbonDataQA2 commented 2 years ago

Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_2.4.5/4654/

CarbonDataQA2 commented 2 years ago

Build Failed with Spark 3.1, Please check CI http://121.244.95.60:12602/job/ApacheCarbon_PR_Builder_3.1/783/

shenjiayu17 commented 1 year ago

Build Failed with Spark 2.4.5, Please check CI http://159.138.8.58:12602/job/ApacheCarbon_PR_Builder_2.4.5/4659/