New Modified YCSB Benchmark

[abhishekjoshi2]

The baseline is still not that great :(

[eric-haibin-lin]

@abhishekjoshi2 Did you set DEFAULT_TUPLES_PER_TILEGROUP to 1000000? Did you observe dropping throughput in the 10 seconds?

[abhishekjoshi2]

DEFAULT_TILEGROUP_SIZE IS 1000000, no difference :(

[eric-haibin-lin]

@jessesleeping @yingjunwu After fixing the random number generation bug, the result is not good

[yingjunwu]

what's the scale factor?

[abhishekjoshi2]

4 4 10 4 0 1 0 32768 0 [0 - 0.1 s]: 139990 0.00078577 [0.1 - 0.2 s]: 127790 0.00101729 [0.2 - 0.3 s]: 120940 0.000826856 [0.3 - 0.4 s]: 117230 0.000938326 [0.4 - 0.5 s]: 113810 0.000790792 [0.5 - 0.6 s]: 109120 0.000641496 [0.6 - 0.7 s]: 109370 0.000640029 [0.7 - 0.8 s]: 105510 0.000663444 [0.8 - 0.9 s]: 103100 0.000872939 [0.9 - 1 s]: 100420 0.00139414 [1 - 1.1 s]: 97430 0.000718465 [1.1 - 1.2 s]: 94390 0.000529717 [1.2 - 1.3 s]: 79700 0.000878294 [1.3 - 1.4 s]: 85760 0.000932836 [1.4 - 1.5 s]: 87900 0 [1.5 - 1.6 s]: 86040 0.000581125 [1.6 - 1.7 s]: 82700 0.00108827 [1.7 - 1.8 s]: 80720 0.00074331 [1.8 - 1.9 s]: 78970 0.00151956 [1.9 - 2 s]: 76720 0.000391032 [2 - 2.1 s]: 75160 0.00133049 [2.1 - 2.2 s]: 73940 0.0012172 [2.2 - 2.3 s]: 72020 0.000555401 [2.3 - 2.4 s]: 70090 0.000428021 [2.4 - 2.5 s]: 68930 0.000290149 [2.5 - 2.6 s]: 67630 0.00044359 [2.6 - 2.7 s]: 66910 0.00149454 [2.7 - 2.8 s]: 66290 0.000754262 [2.8 - 2.9 s]: 64980 0.000307787 [2.9 - 3 s]: 63780 0.00109752 [3 - 3.1 s]: 62650 0.000798085 [3.1 - 3.2 s]: 62540 0.000479693 [3.2 - 3.3 s]: 60380 0.000331236 [3.3 - 3.4 s]: 57040 0.00175316 [3.4 - 3.5 s]: 60040 0.000832778 [3.5 - 3.6 s]: 59220 0.00151976 [3.6 - 3.7 s]: 58770 0.000170155 [3.7 - 3.8 s]: 58300 0.00051458 [3.8 - 3.9 s]: 57520 0.000521558 [3.9 - 4 s]: 57100 0.000350263 [4 - 4.1 s]: 56930 0.00140523 [4.1 - 4.2 s]: 55990 0.00178603 [4.2 - 4.3 s]: 55480 0.00108147 [4.3 - 4.4 s]: 54750 0.000730594 [4.4 - 4.5 s]: 54690 0.000548546 [4.5 - 4.6 s]: 53280 0.000750751 [4.6 - 4.7 s]: 53970 0.000555864 [4.7 - 4.8 s]: 53230 0.000751456 [4.8 - 4.9 s]: 52750 0.000947867 [4.9 - 5 s]: 52330 0.000573285 [5 - 5.1 s]: 51890 0.000770861 [5.1 - 5.2 s]: 51540 0.000582072 [5.2 - 5.3 s]: 50580 0.00059312 [5.3 - 5.4 s]: 50770 0.000393933 [5.4 - 5.5 s]: 50770 0.0005909 [5.5 - 5.6 s]: 50080 0.000399361 [5.6 - 5.7 s]: 49750 0.000603015 [5.7 - 5.8 s]: 49430 0.000809225 [5.8 - 5.9 s]: 49150 0.000813835 [5.9 - 6 s]: 48810 0.00163901 [6 - 6.1 s]: 48310 0.000620989 [6.1 - 6.2 s]: 48000 0.000833333 [6.2 - 6.3 s]: 47760 0.000628141 [6.3 - 6.4 s]: 47700 0.000419287 [6.4 - 6.5 s]: 47350 0.00063358 [6.5 - 6.6 s]: 46670 0.000857082 [6.6 - 6.7 s]: 46910 0.000639522 [6.7 - 6.8 s]: 46490 0.0008604 [6.8 - 6.9 s]: 46490 0.0010755 [6.9 - 7 s]: 46030 0.000868998 [7 - 7.1 s]: 45450 0.000440044 [7.1 - 7.2 s]: 45450 0.00110011 [7.2 - 7.3 s]: 45120 0.000664894 [7.3 - 7.4 s]: 45100 0.000886918 [7.4 - 7.5 s]: 44690 0.000447527 [7.5 - 7.6 s]: 44530 0.000449135 [7.6 - 7.7 s]: 44130 0.00113302 [7.7 - 7.8 s]: 44170 0.000905592 [7.8 - 7.9 s]: 43670 0.00114495 [7.9 - 8 s]: 43700 0.000686499 [8 - 8.1 s]: 43580 0.000917852 [8.1 - 8.2 s]: 42840 0.000466853 [8.2 - 8.3 s]: 42950 0.000232829 [8.3 - 8.4 s]: 43010 0.000697512 [8.4 - 8.5 s]: 41830 0.000478126 [8.5 - 8.6 s]: 37290 0.000804505 [8.6 - 8.7 s]: 37020 0.000810373 [8.7 - 8.8 s]: 37150 0.000269179 [8.8 - 8.9 s]: 39330 0.000508518 [8.9 - 9 s]: 41780 0.00119674 [9 - 9.1 s]: 41870 0.000477669 [9.1 - 9.2 s]: 41420 0.000965717 [9.2 - 9.3 s]: 41230 0.000242542 [9.3 - 9.4 s]: 40310 0.00099231 [9.4 - 9.5 s]: 41060 0.000730638 [9.5 - 9.6 s]: 40750 0.000245399 [9.6 - 9.7 s]: 40530 0.000246731 [9.7 - 9.8 s]: 40490 0.00148185 [9.8 - 9.9 s]: 40440 0.00074184 [9.9 - 10 s]: 40210 0.000746083 60598.8 0.000772293 ~
~
~
~
~

[abhishekjoshi2]

Inserts only Scale factor = number of backends Effects of logging... In a nutshell..

[abhishekjoshi2]

Benchmarking multiple loggers

[yingjunwu]

Interesting numbers. I think the decrease of the throughput with no logging is due to GC. I think you could try to further optimize the synchronized logging (if you still have time :-))

[abhishekjoshi2]

Throughput over time with 24 backends... (no checkpointing)

[eric-haibin-lin]

@yingjunwu We're testing with 100% insert workload here, so GC doesn't help in this case. For synchronous logging, group commit needs to be implemented but that's not within our goal..

[yingjunwu]

ok that's fine. Then you can focus on optimizing asynchronous logging and checkpointing.

If you find that the throughput of SSD is a major bottleneck, then you can try to dump the log to /dev/shm, which refers to the computer's memory. For example, if the original logging dumps the logs to the file /home/yourname/test.log, then now you can try to dump to /dev/shm/test.log. This is merely a simulation but helps you understand what is the real bottleneck.

Remember to clean it up after the experiments though...

I will look at the GC and get back to you later.

[abhishekjoshi2]

Effect of checkpoints.. (no logging)

[eric-haibin-lin]

[yingjunwu]

I don't understand the figure for checkpoints. How many threads did you use for checkpointing? I noticed that you run the workloads for 10 seconds. However, is the time enough for checkpointing the whole database? I think you can learn from Figure 2 in the paper: http://cs-www.cs.yale.edu/homes/dna/papers/fast-checkpoint-sigmod16.pdf.

[MattPerron]

@yingjunwu We are still running expiriments but, the graph we have above is not intending to show how log checkpoint creation takes, only the impact of running a checkpoint during transaction processing.

[yingjunwu]

@MattPerron that's reasonable. I expect the checkpointing will not interfere with the transaction processing, as it is a multi-version system. nice work!