Open GoogleCodeExporter opened 9 years ago
Background: recently, pleased with the performance and lack of errors with LevelDB, we decided to turn on paranoid mode. Paranoid mode ended up being all sorts of bad, as it frequently complained of corruptions (failing to open the entire DB) for evidently pretty common events in the case of process death. We discovered that if you turn on paranoid mode, write tons of data to LevelDB, then kill the process, the log is easily corrupted, and we suspect it's due to mmap flushing on page-aligned boundaries. Below is one such example. The absolute offset inside this file where the data turns into zeros is aligned on a 16k boundary. We suspect this is probably the kernel pushing pages out. It is also the last record in the file. Power loss or some other condition while writing a record could easily cause this, and I am able to reproduce by kill -9'ing a process that is constantly putting records. 00000000 3e 7e f8 f8 43 0c 01 af 4f 04 00 00 00 00 00 01 |>~..C...O.......| 00000010 00 00 00 01 21 02 02 cf af db c0 fb 24 bc be 8c |....!.......$...| 00000020 42 15 95 a3 b1 8b 79 93 8a 3f 86 50 a0 83 49 fa |B.....y..?.P..I.| 00000030 90 cd 22 bb aa 2e 92 18 07 04 df a3 48 1b 00 2e |..".........H...| 00000040 0a 82 16 10 e9 f4 e0 1b 18 00 28 28 30 14 3a 44 |..........((0.:D| 00000050 0a 20 9e c9 16 c7 3b 07 3c 82 ea 76 ae 1d 63 22 |. ....;.<..v..c"| 00000060 68 e3 cd 17 d6 f5 d6 ab 3f a0 6b 29 f3 07 6d 32 |h.......?.k)..m2| 00000070 0e 87 12 20 75 11 ed 58 1d 51 0d e5 13 67 8d b0 |... u..X.Q...g..| 00000080 99 c8 dc ce 53 69 9b 40 bb a3 99 54 a0 83 3e b8 |....Si.@...T..>.| 00000090 59 18 0f b5 3a 44 0a 20 f9 7f da 5b e7 75 1e dc |Y...:D. ...[.u..| 000000a0 bd ad 94 c5 60 94 64 ac 1e 3a 39 4b 12 8d 30 3f |....`.d..:9K..0?| 000000b0 81 27 1d 1e 72 c0 02 55 12 20 5a b9 f0 6f 26 42 |.'..r..U. Z..o&B| 000000c0 8e 9f a5 20 44 82 65 1d 8e 92 2b a6 2d 03 00 f5 |... D.e...+.-...| 000000d0 41 30 a4 68 8e e0 b6 e8 48 7b 3a 44 0a 20 5b 81 |A0.h....H{:D. [.| 000000e0 8b 24 7e de 5d c6 95 d0 f9 d7 66 ae fa cd 06 74 |.$~.].....f....t| 000000f0 9f 85 95 72 82 2d 85 96 8c 8f 84 a3 2e d5 12 20 |...r.-......... | 00000100 d1 bc a9 3d 26 51 28 40 ed 53 bd 9d 7c 62 f6 fc |...=&Q(@.S..|b..| 00000110 01 e4 f6 6c f4 97 d7 87 5f d1 05 44 a7 e9 44 48 |...l...._..D..DH| 00000120 3a 44 0a 20 ca 6e 00 12 26 75 21 a9 f7 b2 dc c5 |:D. .n..&u!.....| 00000130 fb ce 7a df d9 f9 f6 36 98 d7 62 5a a1 85 10 35 |..z....6..bZ...5| 00000140 d9 42 92 77 12 20 a1 cf a9 88 ef fe fd cf 65 92 |.B.w. ........e.| 00000150 1c a6 cc 76 17 2d cd 4e 3f ea 87 3d ef e5 c0 cc |...v.-.N?..=....| 00000160 01 47 27 9f 6a 32 3a 44 0a 20 ac ea e4 8c 82 e2 |.G'.j2:D. ......| 00000170 9a 9e e3 d7 f3 6f 93 55 59 a5 38 30 50 ac 76 c9 |.....o.UY.80P.v.| 00000180 9e 44 b3 9b 25 32 15 6c 17 69 12 20 9d 0e 45 00 |.D..%2.l.i. ..E.| 00000190 aa 54 ee 79 74 56 c2 40 bc 2b 23 c7 c3 c0 62 76 |.T.ytV.@.+#...bv| 000001a0 91 6c 7e 20 97 9b de b3 b2 a1 4f 68 3a 44 0a 20 |.l~ ......Oh:D. | 000001b0 e4 d4 56 d2 d0 53 a7 c1 46 29 68 c5 5e 1a a7 b2 |..V..S..F)h.^...| 000001c0 85 40 eb 34 b5 0c b7 d9 52 fa bc cd 96 93 15 ec |.@.4....R.......| 000001d0 12 20 3c 0d 1f 9f 7f d6 41 24 f3 81 b2 f8 96 84 |. <.....A$......| 000001e0 06 63 10 12 ca c2 20 b0 c6 60 20 9a 8c c1 27 2e |.c.... ..` ...'.| 000001f0 34 91 3a 44 0a 20 cf c6 8b b0 5d 8b 40 41 ae a0 |4.:D. ....].@A..| 00000200 1d f8 1b 75 77 dc 62 f7 d0 6a ba 1e 7a e6 6b 3e |...uw.b..j..z.k>| 00000210 7d 9f 83 b1 07 86 12 20 f4 8d b0 aa 79 70 f1 92 |}...... ....yp..| 00000220 83 d1 2b bd 6d b2 bb b8 54 a7 74 09 12 ea 3d f9 |..+.m...T.t...=.| 00000230 8b be 30 9c bb 6d 28 90 3a 44 0a 20 f2 94 4e c1 |..0..m(.:D. ..N.| 00000240 94 b2 c9 c9 4f 68 4b fa 8c ff 6c 4c 9f 62 82 29 |....OhK...lL.b.)| 00000250 49 75 28 b2 91 68 fb b1 ea e8 29 38 12 20 74 d7 |Iu(..h....)8. t.| 00000260 92 77 97 9d d6 01 63 f4 25 a8 2e 64 88 ee 5e af |.w....c.%..d..^.| 00000270 f8 4e da 6f 60 ae f1 e6 36 72 4f 8a c9 a8 3a 44 |.N.o`...6rO...:D| 00000280 0a 20 10 7a 00 f6 f1 8a 15 1f 92 2a 0b fd 0c 23 |. .z.......*...#| 00000290 db 5c 94 42 3c 14 87 a2 e0 c0 ff 8a b2 5d 1c cb |.\.B<........]..| 000002a0 c4 4a 12 20 15 34 8e 99 6a 9e ee 68 81 a0 36 bb |.J. .4..j..h..6.| 000002b0 04 6e 55 64 2f 51 66 b6 1e 14 44 2e e0 6c 5c 10 |.nUd/Qf...D..l\.| 000002c0 bd e5 6c 26 3a 44 0a 20 56 63 ca 37 de ba bc 62 |..l&:D. Vc.7...b| 000002d0 7e e2 98 68 f7 5e fd a6 27 f4 83 14 47 f4 e1 1f |~..h.^..'...G...| 000002e0 25 b9 95 43 c7 b2 d1 e4 12 20 d4 b4 4f fa 79 cb |%..C..... ..O.y.| 000002f0 a5 e7 39 87 9e b3 cd 17 8a 55 3b 74 17 18 7d 5b |..9......U;t..}[| 00000300 04 c4 3e 15 2d 49 5f dc 9d ae 3a 44 0a 20 28 08 |..>.-I_...:D. (.| 00000310 d4 79 a1 eb b1 cc 4c 8c 2d 02 aa 30 12 ae f3 c3 |.y....L.-..0....| 00000320 12 f6 83 ef 51 0a 54 5e 1a 27 db 15 20 d2 12 20 |....Q.T^.'.. .. | 00000330 13 07 b2 fc 84 fc 79 89 8f 93 70 b5 84 0f c3 21 |......y...p....!| 00000340 80 a7 71 67 c0 25 c3 82 be 5d 52 ec 4b ba 7a f4 |..qg.%...]R.K.z.| 00000350 3a 44 0a 20 91 e3 79 7e 1a 67 4c 58 7b ea 33 ab |:D. ..y~.gLX{.3.| 00000360 be f8 45 c0 f9 ab 6c ed 96 c7 01 0e 91 ae af e1 |..E...l.........| 00000370 52 16 07 36 12 20 2c 08 b2 9a 6d 63 a9 96 3d 11 |R..6. ,...mc..=.| 00000380 55 de 08 9b 85 58 d5 9e 74 d8 32 fd 85 b4 b7 69 |U....X..t.2....i| 00000390 96 9b 10 1a 5d ae 3a 44 0a 20 06 ac 73 31 42 36 |....].:D. ..s1B6| 000003a0 36 66 c3 9f 9b 29 7e e2 39 e8 76 48 42 23 f6 79 |6f...)~.9.vHB#.y| 000003b0 9a a1 10 69 9f 07 ae 24 ec b5 12 20 c6 e1 30 e4 |...i...$... ..0.| 000003c0 19 9e 42 ad b1 62 35 59 5b cf 04 99 dc 67 14 84 |..B..b5Y[....g..| 000003d0 32 c8 9b be 0c 23 af 92 57 85 56 c2 3a 44 0a 20 |2....#..W.V.:D. | 000003e0 1a 7d d6 d0 c9 f7 d2 32 21 2b c4 a6 69 37 4c 01 |.}.....2!+..i7L.| 000003f0 ed db cd 2b e4 a6 3d f8 3e 1c 8b 91 b6 d8 3b 4b |...+..=.>.....;K| 00000400 12 20 6a ee fe 34 77 d3 37 22 9c c0 26 cb f6 36 |. j..4w.7"..&..6| 00000410 ea 54 10 0d 0c f6 01 9b 90 3e c0 d2 13 93 d8 c3 |.T.......>......| 00000420 6b 69 3a 44 0a 20 ce e6 55 42 9c 50 91 f3 dc 57 |ki:D. ..UB.P...W| 00000430 ff 96 7d 9b fb 76 68 35 e3 73 1f 5c c1 c9 3b 8c |..}..vh5.s.\..;.| 00000440 d7 79 56 0b 47 24 12 20 4f 9e c7 c7 3f b6 5e fb |.yV.G$. O...?.^.| 00000450 e6 7b 54 7c 8c 6c 81 f8 26 e9 31 87 17 75 d1 53 |.{T|.l..&.1..u.S| 00000460 3e 19 c9 a8 6d 11 47 26 3a 44 0a 20 14 24 f2 0c |>...m.G&:D. .$..| 00000470 1e 20 3f 7c 45 a0 e3 28 a0 c8 66 08 a6 c3 8f 64 |. ?|E..(..f....d| 00000480 45 7a dc ee ae e7 a2 b4 66 cc 6d d6 12 20 5b 83 |Ez......f.m.. [.| 00000490 ab 45 12 f8 ca f2 a8 d8 f9 c3 5a 1a e3 80 d1 fc |.E........Z.....| 000004a0 15 98 f8 5a e4 39 4e bc 35 28 d1 c9 fe 25 3a 44 |...Z.9N.5(...%:D| 000004b0 0a 20 74 49 c2 4b ce 2b 26 c7 c6 d8 d6 61 c9 bd |. tI.K.+&....a..| 000004c0 39 49 11 91 6f 7d 31 4a 4d 39 6f 7d 3f 29 e6 6b |9I..o}1JM9o}?).k| 000004d0 c0 e5 12 20 1f ad 1e b9 b9 9c da 2b 40 90 a0 45 |... .......+@..E| 000004e0 eb ad 95 1e 71 a7 ca 9e 2d b8 8e 23 e6 a6 cf 17 |....q...-..#....| 000004f0 23 f2 06 ba 3a 44 0a 20 2f 2d 0e bb cb d2 d1 c9 |#...:D. /-......| 00000500 1b 91 42 d7 3a 17 f1 1d 63 8d fc fb be 32 54 2b |..B.:...c....2T+| 00000510 20 6c ea c6 02 ba 3b 42 12 20 65 33 1b ff 02 5e | l....;B. e3...^| 00000520 80 22 ef 75 fa 38 5e f4 b7 67 37 70 6c 74 8a 36 |.".u.8^..g7plt.6| 00000530 1c 07 cb 0b 18 b3 96 15 21 54 3a 44 0a 20 b6 3d |........!T:D. .=| 00000540 ec 7c 36 e9 1c 4c 4f f7 99 9f d5 1b bb 96 10 04 |.|6..LO.........| 00000550 88 8f 35 b7 04 44 aa 35 48 b1 c1 8c 75 26 12 20 |..5..D.5H...u&. | 00000560 87 bf 03 56 b5 56 1c c3 03 1f ba f7 e0 5f c1 f3 |...V.V......._..| 00000570 5a 78 dc e2 b9 aa 05 25 6e a0 1d 6c f7 13 fc 4a |Zx.....%n..l...J| 00000580 3a 44 0a 20 94 d8 c5 d5 2e e9 9d ee 48 1c 5d a7 |:D. ........H.].| 00000590 fc fa 5c 20 3f f5 09 da 28 75 c4 ad df 88 b5 f2 |..\ ?...(u......| 000005a0 fe ff e5 3f 12 20 43 5a a5 b2 32 c8 cc 69 a4 51 |...?. CZ..2..i.Q| 000005b0 e6 e9 1b c5 72 36 35 e6 99 29 0f 13 01 d9 ea c2 |....r65..)......| 000005c0 08 ca 39 a3 7b c8 3a 44 0a 20 bb 44 c3 ad f1 38 |..9.{.:D. .D...8| 000005d0 68 ef 3b 66 f3 0a ff 7c 0d c8 d4 70 4c 20 f3 68 |h.;f...|...pL .h| 000005e0 88 9a c2 2d 2b 24 a8 98 ff 19 12 20 bd 75 33 7b |...-+$..... .u3{| 000005f0 c5 a3 2b 91 f5 59 df 3d 25 69 a6 50 a1 ee c7 21 |..+..Y.=%i.P...!| 00000600 3e c2 b2 5a c3 24 0d ac 4c 25 01 26 3a 44 0a 20 |>..Z.$..L%.&:D. | 00000610 43 20 4a 7c 87 3f a0 2c 6c 3a d7 2d 5e 32 f8 1c |C J|.?.,l:.-^2..| 00000620 7d 62 cf 07 7e 9e 3e 99 09 d8 34 0b 27 51 d1 ee |}b..~.>...4.'Q..| 00000630 12 20 16 19 2a b3 92 cd 09 26 80 ed 6b e8 03 1e |. ..*....&..k...| 00000640 f8 22 57 59 f2 2c c3 87 28 ff 5d be 14 3f c7 79 |."WY.,..(.]..?.y| 00000650 41 99 3a 44 0a 20 f2 37 46 c9 a9 71 44 16 19 50 |A.:D. .7F..qD..P| 00000660 33 40 47 37 06 2c c1 2e 19 9b fe 8f 21 97 8a fc |3@G7.,......!...| 00000670 b1 72 c2 e9 4d b2 12 20 ed da ed e0 29 ed 67 1a |.r..M.. ....).g.| 00000680 ef 58 9a c6 eb 34 14 e3 1f f2 e5 18 8d 99 a4 ea |.X...4..........| 00000690 e1 40 4f f2 7e 5c bf 63 3a 44 0a 20 ce b1 ee 7a |.@O.~\.c:D. ...z| 000006a0 89 2a e3 c5 01 12 f9 b9 09 44 ad 20 b9 9b b2 34 |.*.......D. ...4| 000006b0 4f c6 41 e1 1e 86 2f b5 f4 60 2c 77 12 20 5e 48 |O.A.../..`,w. ^H| 000006c0 66 ed 22 b0 f0 7b a3 49 0c f9 ef 0d 44 bd 18 4f |f."..{.I....D..O| 000006d0 e0 de a3 48 52 fb 62 dc 8e 87 68 59 77 1f 3a 44 |...HR.b...hYw.:D| 000006e0 0a 20 30 e8 b8 e1 4c 4d e7 f9 22 95 d6 e7 87 e8 |. 0...LM..".....| 000006f0 c1 8e 1d 7e 11 ac 32 9f 3e 7e c3 9a 64 39 69 b5 |...~..2.>~..d9i.| 00000700 1a 67 12 20 ac 25 37 86 47 28 03 d4 21 df 33 b4 |.g. .%7.G(..!.3.| 00000710 1a 9e f7 c7 81 ca 17 81 ec be 4e 6b 53 87 4b 11 |..........NkS.K.| 00000720 c7 da 62 6a 3a 44 0a 20 42 da c8 34 d2 f5 02 69 |..bj:D. B..4...i| 00000730 5e 0e 7a e1 a1 65 0f 72 38 3f ef b5 04 ae e3 98 |^.z..e.r8?......| 00000740 06 cc e2 a6 4b e6 c8 89 12 20 f8 92 28 cc 3c fa |....K.... ..(.<.| 00000750 8b 60 e6 9c 93 a5 d1 84 5d 6a da 10 30 7f 99 3d |.`......]j..0..=| 00000760 23 8a e7 44 95 44 87 fe 83 d2 3a 44 0a 20 64 65 |#..D.D....:D. de| 00000770 f1 27 69 b2 bd c5 83 32 74 46 75 ea 24 4e 57 7f |.'i....2tFu.$NW.| 00000780 74 80 99 91 cd 2f ab 99 b0 3e 46 12 1e d1 12 20 |t..../...>F.... | 00000790 d7 b7 b1 4d 11 a9 57 25 35 a0 3c a4 5e 95 8c 00 |...M..W%5.<.^...| 000007a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000007b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000007c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| // // zeros continue until end of record, and rest of the file. this is the last block // I feel like paranoid mode should still work? Or is paranoid mode not recommended for production settings?
Original issue reported on code.google.com by jtolds on 29 Jul 2013 at 3:46
jtolds
I think LevelDB should not use mmap, it is easy to lost data on power failure, process failure, and etc.
Original comment by wuzuy...@gmail.com on 16 Sep 2013 at 4:59
wuzuy...@gmail.com
Original issue reported on code.google.com by
jtolds
on 29 Jul 2013 at 3:46