search

chrsmithdemos / leveldb

Automatically exported from code.google.com/p/leveldb
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

LevelDB corruption (mmap flush behavior?) #196

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
Background: recently, pleased with the performance and lack of errors with 
LevelDB, we decided to turn on paranoid mode. Paranoid mode ended up being all 
sorts of bad, as it frequently complained of corruptions (failing to open the 
entire DB) for evidently pretty common events in the case of process death.

We discovered that if you turn on paranoid mode, write tons of data to LevelDB, 
then kill the process, the log is easily corrupted, and we suspect it's due to 
mmap flushing on page-aligned boundaries.

Below is one such example.

The absolute offset inside this file where the data turns into zeros is
aligned on a 16k boundary. We suspect this is probably the kernel pushing pages 
out. It is also the last record in the file. Power loss or some other condition 
while writing a record could easily cause this, and I am able to reproduce by 
kill -9'ing a process that is constantly putting records.

    00000000  3e 7e f8 f8 43 0c 01 af  4f 04 00 00 00 00 00 01  |>~..C...O.......|
    00000010  00 00 00 01 21 02 02 cf  af db c0 fb 24 bc be 8c  |....!.......$...|
    00000020  42 15 95 a3 b1 8b 79 93  8a 3f 86 50 a0 83 49 fa  |B.....y..?.P..I.|
    00000030  90 cd 22 bb aa 2e 92 18  07 04 df a3 48 1b 00 2e  |..".........H...|
    00000040  0a 82 16 10 e9 f4 e0 1b  18 00 28 28 30 14 3a 44  |..........((0.:D|
    00000050  0a 20 9e c9 16 c7 3b 07  3c 82 ea 76 ae 1d 63 22  |. ....;.<..v..c"|
    00000060  68 e3 cd 17 d6 f5 d6 ab  3f a0 6b 29 f3 07 6d 32  |h.......?.k)..m2|
    00000070  0e 87 12 20 75 11 ed 58  1d 51 0d e5 13 67 8d b0  |... u..X.Q...g..|
    00000080  99 c8 dc ce 53 69 9b 40  bb a3 99 54 a0 83 3e b8  |....Si.@...T..>.|
    00000090  59 18 0f b5 3a 44 0a 20  f9 7f da 5b e7 75 1e dc  |Y...:D. ...[.u..|
    000000a0  bd ad 94 c5 60 94 64 ac  1e 3a 39 4b 12 8d 30 3f  |....`.d..:9K..0?|
    000000b0  81 27 1d 1e 72 c0 02 55  12 20 5a b9 f0 6f 26 42  |.'..r..U. Z..o&B|
    000000c0  8e 9f a5 20 44 82 65 1d  8e 92 2b a6 2d 03 00 f5  |... D.e...+.-...|
    000000d0  41 30 a4 68 8e e0 b6 e8  48 7b 3a 44 0a 20 5b 81  |A0.h....H{:D. [.|
    000000e0  8b 24 7e de 5d c6 95 d0  f9 d7 66 ae fa cd 06 74  |.$~.].....f....t|
    000000f0  9f 85 95 72 82 2d 85 96  8c 8f 84 a3 2e d5 12 20  |...r.-......... |
    00000100  d1 bc a9 3d 26 51 28 40  ed 53 bd 9d 7c 62 f6 fc  |...=&Q(@.S..|b..|
    00000110  01 e4 f6 6c f4 97 d7 87  5f d1 05 44 a7 e9 44 48  |...l...._..D..DH|
    00000120  3a 44 0a 20 ca 6e 00 12  26 75 21 a9 f7 b2 dc c5  |:D. .n..&u!.....|
    00000130  fb ce 7a df d9 f9 f6 36  98 d7 62 5a a1 85 10 35  |..z....6..bZ...5|
    00000140  d9 42 92 77 12 20 a1 cf  a9 88 ef fe fd cf 65 92  |.B.w. ........e.|
    00000150  1c a6 cc 76 17 2d cd 4e  3f ea 87 3d ef e5 c0 cc  |...v.-.N?..=....|
    00000160  01 47 27 9f 6a 32 3a 44  0a 20 ac ea e4 8c 82 e2  |.G'.j2:D. ......|
    00000170  9a 9e e3 d7 f3 6f 93 55  59 a5 38 30 50 ac 76 c9  |.....o.UY.80P.v.|
    00000180  9e 44 b3 9b 25 32 15 6c  17 69 12 20 9d 0e 45 00  |.D..%2.l.i. ..E.|
    00000190  aa 54 ee 79 74 56 c2 40  bc 2b 23 c7 c3 c0 62 76  |.T.ytV.@.+#...bv|
    000001a0  91 6c 7e 20 97 9b de b3  b2 a1 4f 68 3a 44 0a 20  |.l~ ......Oh:D. |
    000001b0  e4 d4 56 d2 d0 53 a7 c1  46 29 68 c5 5e 1a a7 b2  |..V..S..F)h.^...|
    000001c0  85 40 eb 34 b5 0c b7 d9  52 fa bc cd 96 93 15 ec  |.@.4....R.......|
    000001d0  12 20 3c 0d 1f 9f 7f d6  41 24 f3 81 b2 f8 96 84  |. <.....A$......|
    000001e0  06 63 10 12 ca c2 20 b0  c6 60 20 9a 8c c1 27 2e  |.c.... ..` ...'.|
    000001f0  34 91 3a 44 0a 20 cf c6  8b b0 5d 8b 40 41 ae a0  |4.:D. ....].@A..|
    00000200  1d f8 1b 75 77 dc 62 f7  d0 6a ba 1e 7a e6 6b 3e  |...uw.b..j..z.k>|
    00000210  7d 9f 83 b1 07 86 12 20  f4 8d b0 aa 79 70 f1 92  |}...... ....yp..|
    00000220  83 d1 2b bd 6d b2 bb b8  54 a7 74 09 12 ea 3d f9  |..+.m...T.t...=.|
    00000230  8b be 30 9c bb 6d 28 90  3a 44 0a 20 f2 94 4e c1  |..0..m(.:D. ..N.|
    00000240  94 b2 c9 c9 4f 68 4b fa  8c ff 6c 4c 9f 62 82 29  |....OhK...lL.b.)|
    00000250  49 75 28 b2 91 68 fb b1  ea e8 29 38 12 20 74 d7  |Iu(..h....)8. t.|
    00000260  92 77 97 9d d6 01 63 f4  25 a8 2e 64 88 ee 5e af  |.w....c.%..d..^.|
    00000270  f8 4e da 6f 60 ae f1 e6  36 72 4f 8a c9 a8 3a 44  |.N.o`...6rO...:D|
    00000280  0a 20 10 7a 00 f6 f1 8a  15 1f 92 2a 0b fd 0c 23  |. .z.......*...#|
    00000290  db 5c 94 42 3c 14 87 a2  e0 c0 ff 8a b2 5d 1c cb  |.\.B<........]..|
    000002a0  c4 4a 12 20 15 34 8e 99  6a 9e ee 68 81 a0 36 bb  |.J. .4..j..h..6.|
    000002b0  04 6e 55 64 2f 51 66 b6  1e 14 44 2e e0 6c 5c 10  |.nUd/Qf...D..l\.|
    000002c0  bd e5 6c 26 3a 44 0a 20  56 63 ca 37 de ba bc 62  |..l&:D. Vc.7...b|
    000002d0  7e e2 98 68 f7 5e fd a6  27 f4 83 14 47 f4 e1 1f  |~..h.^..'...G...|
    000002e0  25 b9 95 43 c7 b2 d1 e4  12 20 d4 b4 4f fa 79 cb  |%..C..... ..O.y.|
    000002f0  a5 e7 39 87 9e b3 cd 17  8a 55 3b 74 17 18 7d 5b  |..9......U;t..}[|
    00000300  04 c4 3e 15 2d 49 5f dc  9d ae 3a 44 0a 20 28 08  |..>.-I_...:D. (.|
    00000310  d4 79 a1 eb b1 cc 4c 8c  2d 02 aa 30 12 ae f3 c3  |.y....L.-..0....|
    00000320  12 f6 83 ef 51 0a 54 5e  1a 27 db 15 20 d2 12 20  |....Q.T^.'.. .. |
    00000330  13 07 b2 fc 84 fc 79 89  8f 93 70 b5 84 0f c3 21  |......y...p....!|
    00000340  80 a7 71 67 c0 25 c3 82  be 5d 52 ec 4b ba 7a f4  |..qg.%...]R.K.z.|
    00000350  3a 44 0a 20 91 e3 79 7e  1a 67 4c 58 7b ea 33 ab  |:D. ..y~.gLX{.3.|
    00000360  be f8 45 c0 f9 ab 6c ed  96 c7 01 0e 91 ae af e1  |..E...l.........|
    00000370  52 16 07 36 12 20 2c 08  b2 9a 6d 63 a9 96 3d 11  |R..6. ,...mc..=.|
    00000380  55 de 08 9b 85 58 d5 9e  74 d8 32 fd 85 b4 b7 69  |U....X..t.2....i|
    00000390  96 9b 10 1a 5d ae 3a 44  0a 20 06 ac 73 31 42 36  |....].:D. ..s1B6|
    000003a0  36 66 c3 9f 9b 29 7e e2  39 e8 76 48 42 23 f6 79  |6f...)~.9.vHB#.y|
    000003b0  9a a1 10 69 9f 07 ae 24  ec b5 12 20 c6 e1 30 e4  |...i...$... ..0.|
    000003c0  19 9e 42 ad b1 62 35 59  5b cf 04 99 dc 67 14 84  |..B..b5Y[....g..|
    000003d0  32 c8 9b be 0c 23 af 92  57 85 56 c2 3a 44 0a 20  |2....#..W.V.:D. |
    000003e0  1a 7d d6 d0 c9 f7 d2 32  21 2b c4 a6 69 37 4c 01  |.}.....2!+..i7L.|
    000003f0  ed db cd 2b e4 a6 3d f8  3e 1c 8b 91 b6 d8 3b 4b  |...+..=.>.....;K|
    00000400  12 20 6a ee fe 34 77 d3  37 22 9c c0 26 cb f6 36  |. j..4w.7"..&..6|
    00000410  ea 54 10 0d 0c f6 01 9b  90 3e c0 d2 13 93 d8 c3  |.T.......>......|
    00000420  6b 69 3a 44 0a 20 ce e6  55 42 9c 50 91 f3 dc 57  |ki:D. ..UB.P...W|
    00000430  ff 96 7d 9b fb 76 68 35  e3 73 1f 5c c1 c9 3b 8c  |..}..vh5.s.\..;.|
    00000440  d7 79 56 0b 47 24 12 20  4f 9e c7 c7 3f b6 5e fb  |.yV.G$. O...?.^.|
    00000450  e6 7b 54 7c 8c 6c 81 f8  26 e9 31 87 17 75 d1 53  |.{T|.l..&.1..u.S|
    00000460  3e 19 c9 a8 6d 11 47 26  3a 44 0a 20 14 24 f2 0c  |>...m.G&:D. .$..|
    00000470  1e 20 3f 7c 45 a0 e3 28  a0 c8 66 08 a6 c3 8f 64  |. ?|E..(..f....d|
    00000480  45 7a dc ee ae e7 a2 b4  66 cc 6d d6 12 20 5b 83  |Ez......f.m.. [.|
    00000490  ab 45 12 f8 ca f2 a8 d8  f9 c3 5a 1a e3 80 d1 fc  |.E........Z.....|
    000004a0  15 98 f8 5a e4 39 4e bc  35 28 d1 c9 fe 25 3a 44  |...Z.9N.5(...%:D|
    000004b0  0a 20 74 49 c2 4b ce 2b  26 c7 c6 d8 d6 61 c9 bd  |. tI.K.+&....a..|
    000004c0  39 49 11 91 6f 7d 31 4a  4d 39 6f 7d 3f 29 e6 6b  |9I..o}1JM9o}?).k|
    000004d0  c0 e5 12 20 1f ad 1e b9  b9 9c da 2b 40 90 a0 45  |... .......+@..E|
    000004e0  eb ad 95 1e 71 a7 ca 9e  2d b8 8e 23 e6 a6 cf 17  |....q...-..#....|
    000004f0  23 f2 06 ba 3a 44 0a 20  2f 2d 0e bb cb d2 d1 c9  |#...:D. /-......|
    00000500  1b 91 42 d7 3a 17 f1 1d  63 8d fc fb be 32 54 2b  |..B.:...c....2T+|
    00000510  20 6c ea c6 02 ba 3b 42  12 20 65 33 1b ff 02 5e  | l....;B. e3...^|
    00000520  80 22 ef 75 fa 38 5e f4  b7 67 37 70 6c 74 8a 36  |.".u.8^..g7plt.6|
    00000530  1c 07 cb 0b 18 b3 96 15  21 54 3a 44 0a 20 b6 3d  |........!T:D. .=|
    00000540  ec 7c 36 e9 1c 4c 4f f7  99 9f d5 1b bb 96 10 04  |.|6..LO.........|
    00000550  88 8f 35 b7 04 44 aa 35  48 b1 c1 8c 75 26 12 20  |..5..D.5H...u&. |
    00000560  87 bf 03 56 b5 56 1c c3  03 1f ba f7 e0 5f c1 f3  |...V.V......._..|
    00000570  5a 78 dc e2 b9 aa 05 25  6e a0 1d 6c f7 13 fc 4a  |Zx.....%n..l...J|
    00000580  3a 44 0a 20 94 d8 c5 d5  2e e9 9d ee 48 1c 5d a7  |:D. ........H.].|
    00000590  fc fa 5c 20 3f f5 09 da  28 75 c4 ad df 88 b5 f2  |..\ ?...(u......|
    000005a0  fe ff e5 3f 12 20 43 5a  a5 b2 32 c8 cc 69 a4 51  |...?. CZ..2..i.Q|
    000005b0  e6 e9 1b c5 72 36 35 e6  99 29 0f 13 01 d9 ea c2  |....r65..)......|
    000005c0  08 ca 39 a3 7b c8 3a 44  0a 20 bb 44 c3 ad f1 38  |..9.{.:D. .D...8|
    000005d0  68 ef 3b 66 f3 0a ff 7c  0d c8 d4 70 4c 20 f3 68  |h.;f...|...pL .h|
    000005e0  88 9a c2 2d 2b 24 a8 98  ff 19 12 20 bd 75 33 7b  |...-+$..... .u3{|
    000005f0  c5 a3 2b 91 f5 59 df 3d  25 69 a6 50 a1 ee c7 21  |..+..Y.=%i.P...!|
    00000600  3e c2 b2 5a c3 24 0d ac  4c 25 01 26 3a 44 0a 20  |>..Z.$..L%.&:D. |
    00000610  43 20 4a 7c 87 3f a0 2c  6c 3a d7 2d 5e 32 f8 1c  |C J|.?.,l:.-^2..|
    00000620  7d 62 cf 07 7e 9e 3e 99  09 d8 34 0b 27 51 d1 ee  |}b..~.>...4.'Q..|
    00000630  12 20 16 19 2a b3 92 cd  09 26 80 ed 6b e8 03 1e  |. ..*....&..k...|
    00000640  f8 22 57 59 f2 2c c3 87  28 ff 5d be 14 3f c7 79  |."WY.,..(.]..?.y|
    00000650  41 99 3a 44 0a 20 f2 37  46 c9 a9 71 44 16 19 50  |A.:D. .7F..qD..P|
    00000660  33 40 47 37 06 2c c1 2e  19 9b fe 8f 21 97 8a fc  |3@G7.,......!...|
    00000670  b1 72 c2 e9 4d b2 12 20  ed da ed e0 29 ed 67 1a  |.r..M.. ....).g.|
    00000680  ef 58 9a c6 eb 34 14 e3  1f f2 e5 18 8d 99 a4 ea  |.X...4..........|
    00000690  e1 40 4f f2 7e 5c bf 63  3a 44 0a 20 ce b1 ee 7a  |.@O.~\.c:D. ...z|
    000006a0  89 2a e3 c5 01 12 f9 b9  09 44 ad 20 b9 9b b2 34  |.*.......D. ...4|
    000006b0  4f c6 41 e1 1e 86 2f b5  f4 60 2c 77 12 20 5e 48  |O.A.../..`,w. ^H|
    000006c0  66 ed 22 b0 f0 7b a3 49  0c f9 ef 0d 44 bd 18 4f  |f."..{.I....D..O|
    000006d0  e0 de a3 48 52 fb 62 dc  8e 87 68 59 77 1f 3a 44  |...HR.b...hYw.:D|
    000006e0  0a 20 30 e8 b8 e1 4c 4d  e7 f9 22 95 d6 e7 87 e8  |. 0...LM..".....|
    000006f0  c1 8e 1d 7e 11 ac 32 9f  3e 7e c3 9a 64 39 69 b5  |...~..2.>~..d9i.|
    00000700  1a 67 12 20 ac 25 37 86  47 28 03 d4 21 df 33 b4  |.g. .%7.G(..!.3.|
    00000710  1a 9e f7 c7 81 ca 17 81  ec be 4e 6b 53 87 4b 11  |..........NkS.K.|
    00000720  c7 da 62 6a 3a 44 0a 20  42 da c8 34 d2 f5 02 69  |..bj:D. B..4...i|
    00000730  5e 0e 7a e1 a1 65 0f 72  38 3f ef b5 04 ae e3 98  |^.z..e.r8?......|
    00000740  06 cc e2 a6 4b e6 c8 89  12 20 f8 92 28 cc 3c fa  |....K.... ..(.<.|
    00000750  8b 60 e6 9c 93 a5 d1 84  5d 6a da 10 30 7f 99 3d  |.`......]j..0..=|
    00000760  23 8a e7 44 95 44 87 fe  83 d2 3a 44 0a 20 64 65  |#..D.D....:D. de|
    00000770  f1 27 69 b2 bd c5 83 32  74 46 75 ea 24 4e 57 7f  |.'i....2tFu.$NW.|
    00000780  74 80 99 91 cd 2f ab 99  b0 3e 46 12 1e d1 12 20  |t..../...>F.... |
    00000790  d7 b7 b1 4d 11 a9 57 25  35 a0 3c a4 5e 95 8c 00  |...M..W%5.<.^...|
    000007a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
    000007b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
    000007c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
    //
    // zeros continue until end of record, and rest of the file. this is the last block
    //

I feel like paranoid mode should still work? Or is paranoid mode not 
recommended for production settings?

Original issue reported on code.google.com by jtolds on 29 Jul 2013 at 3:46

GoogleCodeExporter commented 9 years ago 
I think LevelDB should not use mmap, it is easy to lost data on power failure, 
process failure, and etc.

Original comment by wuzuy...@gmail.com on 16 Sep 2013 at 4:59