search

xuchuanyin / workbench

0 stars 0 forks source link

2018-11-05 compaction performance #75

Open xuchuanyin opened 5 years ago

xuchuanyin commented 5 years ago

Prefetch disabled

15million, 1.4GB

+-----------------+---------+-----------------------+-----------------------+---------+-----------+---------+----------+
|SegmentSequenceId|Status   |Load Start Time        |Load End Time          |Merged To|File Format|Data Size|Index Size|
+-----------------+---------+-----------------------+-----------------------+---------+-----------+---------+----------+
|2                |Compacted|2018-11-04 23:29:50.029|2018-11-04 23:30:45.128|0.1      |COLUMNAR_V3|1.42GB   |24.44KB   |
|1                |Compacted|2018-11-04 23:28:54.618|2018-11-04 23:29:49.811|0.1      |COLUMNAR_V3|1.42GB   |24.44KB   |
|0.1              |Success  |2018-11-04 23:30:46.782|2018-11-04 23:32:33.907|NA       |COLUMNAR_V3|1.86GB   |33.36KB   |
|0                |Compacted|2018-11-04 23:27:49.769|2018-11-04 23:28:54.306|0.1      |COLUMNAR_V3|1.42GB   |24.44KB   |
+-----------------+---------+-----------------------+-----------------------+---------+-----------+---------+----------+
xuchuanyin commented 5 years ago

Prefetch Enabled

15million, 1.4GB

+-----------------+---------+-----------------------+-----------------------+---------+-----------+---------+----------+
|SegmentSequenceId|Status   |Load Start Time        |Load End Time          |Merged To|File Format|Data Size|Index Size|
+-----------------+---------+-----------------------+-----------------------+---------+-----------+---------+----------+
|2                |Compacted|2018-11-04 23:37:07.999|2018-11-04 23:38:06.033|0.1      |COLUMNAR_V3|1.42GB   |24.44KB   |
|1                |Compacted|2018-11-04 23:36:13.039|2018-11-04 23:37:07.749|0.1      |COLUMNAR_V3|1.42GB   |24.44KB   |
|0.1              |Success  |2018-11-04 23:38:07.858|2018-11-04 23:39:43.174|NA       |COLUMNAR_V3|1.86GB   |33.36KB   |
|0                |Compacted|2018-11-04 23:35:06.989|2018-11-04 23:36:12.649|0.1      |COLUMNAR_V3|1.42GB   |24.44KB   |
+-----------------+---------+-----------------------+-----------------------+---------+-----------+---------+----------+
xuchuanyin commented 5 years ago

Prefetch Disabled

7.3GB lineitem, 60 Million

LOAD: 141.2s, 137.0s, 138.9s
COMPACT: 206.4s

Prefetch Enabled

LOAD: 141.9s, 142.8s, 139.8s
COMPACT: 198.5s
xuchuanyin commented 5 years ago

Prefetch Disabled

carbon.detail.batch.size= 32000 : default is 100

7.3GB lineitem, 60 Million

LOAD: 144.7s, 138.0s, 138.8s
COMPACT: 192.7s

Prefetch Enabled

LOAD: 142.0s, 137.7s, 138.1s
COMPACT: 180s/175.9s/177.3s
xuchuanyin commented 5 years ago

image