syndtr
commented
8 years ago Other than the seventh power of 2, none that I can think of. Do you have the DB file that I can inspect?
Also, issue #128 \o/
Yeah, what's with this number 128 anyway...
Open calmh opened 8 years ago
syndtr
commented
8 years ago Other than the seventh power of 2, none that I can think of. Do you have the DB file that I can inspect?
Also, issue #128 \o/
Yeah, what's with this number 128 anyway...
calmh
commented
8 years ago Yep, here:
https://nym.se/t/index-v0.11.0.db.rar
This is what I see, and I guess if that is what is actually on disk it's just going to be unclear how the zeroes got there:
Key:
00000000 00 4c 6a 75 64 62 c3 b6 63 6b 65 72 00 00 00 00 |.Ljudb..cker....|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000040 00 60 14 db fc f4 22 e6 dc d2 ee 6b 9d 4f 58 f1 |.`...."....k.OX.|
00000050 c8 ae c7 bb ed bb 4a 7e 1b 64 b5 31 cb 6e 74 e0 |......J~.d.1.nt.|
00000060 02 41 74 6c 61 73 20 53 68 72 75 67 67 65 64 20 |.Atlas Shrugged |
00000070 2d 41 79 6e 20 52 61 6e 64 2f 30 39 33 34 20 2d |-Ayn Rand/0934 -|
00000080 33 39 74 2e 6d 70 33 |39t.mp3|
Value:
00000000 00 00 00 26 41 74 6c 61 73 20 53 68 72 75 67 67 |...&Atlas Shrugg|
00000010 65 64 20 2d 41 79 6e 20 52 61 6e 64 2f 30 39 33 |ed -Ayn Rand/093|
00000020 34 20 2d 33 39 74 2e 6d 70 33 00 00 00 00 01 a4 |4 -39t.mp3......|
00000030 00 00 00 00 56 07 01 65 00 00 00 01 07 19 0a e0 |....V..e........|
00000040 af d3 c2 f9 00 00 00 00 00 00 00 01 00 00 00 00 |................|
00000050 00 00 85 f9 00 00 00 19 00 02 00 00 00 00 00 20 |............... |
00000060 38 c2 72 0c a5 15 90 72 bc 2e 04 ee d6 f0 3f 0d |8.r....r......?.|
00000070 84 b5 2b a6 18 f0 90 dd f3 88 18 13 16 0b 4f 20 |..+...........O |
// begin oddity
00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
// end oddity
00000100 b4 af 4d e6 75 a9 b5 9b 61 c4 1a 12 1c 6b 16 28 |..M.u...a....k.(|
00000110 6a 71 77 f5 3c 19 65 be 4c a8 54 ca 93 40 93 75 |jqw.<.e.L.T..@.u|
00000120 00 02 00 00 00 00 00 20 9e 06 69 0e 82 c8 57 0e |....... ..i...W.|
00000130 f9 be b4 57 f1 82 f6 32 4d f1 a5 eb b3 38 a8 21 |...W...2M....8.!|
00000140 ad 71 ce b1 a6 38 24 06 00 02 00 00 00 00 00 20 |.q...8$........ |
00000150 e1 0e be 92 02 0b 30 25 67 e8 b4 00 dd ba 08 b4 |......0%g.......|
00000160 ae f4 88 da f3 e5 96 99 1a 2b db 0d c2 6d bb 08 |.........+...m..|
00000170 00 02 00 00 00 00 00 20 34 d5 b2 54 80 f1 32 55 |....... 4..T..2U|
00000180 eb bc 21 08 ad ab 96 8c 47 2c ca ee 55 b4 88 06 |..!.....G,..U...|
00000190 76 d0 ff ce 28 a4 1f ac 00 02 00 00 00 00 00 20 |v...(.......... |
000001a0 06 f0 a6 9f c0 2f 8f a6 71 89 00 27 30 b2 fe ad |...../..q..'0...|
000001b0 54 19 ed 97 dc 00 18 51 ac 9c 44 35 95 ab d8 30 |T......Q..D5...0|
000001c0 00 02 00 00 00 00 00 20 7a 59 f9 86 4e d4 44 03 |....... zY..N.D.|
000001d0 c7 df fc dc a4 67 65 be cc ec 95 c3 3e 94 46 25 |.....ge.....>.F%|
000001e0 57 e1 59 ed a5 83 58 be 00 02 00 00 00 00 00 20 |W.Y...X........ |
000001f0 20 ce 37 f3 9c 06 f2 88 7a 89 74 58 82 9d ed 02 | .7.....z.tX....|
00000200 1e 30 e6 9d 41 bd 0d 3f ac 8e ed 83 76 93 17 90 |.0..A..?....v...|
00000210 00 02 00 00 00 00 00 20 26 1c 56 16 dc 6f 6d 0e |....... &.V..om.|
00000220 d1 b1 ca 56 4c 14 8c 33 5b bf fa f2 00 e5 33 38 |...VL..3[.....38|
00000230 38 5c 12 e1 86 7d 3a a6 00 02 00 00 00 00 00 20 |8\...}:........ |
00000240 25 28 98 d3 4e 30 84 40 ef 06 38 74 d4 49 85 52 |%(..N0.@..8t.I.R|
00000250 a7 7d 00 ac 84 f2 4d e7 57 ed 01 1d c7 dc 8d 8a |.}....M.W.......|
00000260 00 02 00 00 00 00 00 20 07 f2 c2 7f 79 1d 25 09 |....... ....y.%.|
00000270 8b 24 a2 c4 1a 09 e8 ff 7a d4 d6 99 6c cd b0 10 |.$......z...l...|
00000280 96 07 f0 59 f7 66 ce 5c 00 02 00 00 00 00 00 20 |...Y.f.\....... |
00000290 75 b5 9b 8d 21 fc 23 94 b0 6e 0c a3 be 3b 11 cd |u...!.#..n...;..|
000002a0 d7 66 91 b5 d5 7f 76 75 b5 e0 d1 a7 21 7f 40 93 |.f....vu....!.@.|
000002b0 00 02 00 00 00 00 00 20 1e e1 f6 d5 5c 30 9e af |....... ....\0..|
000002c0 78 67 a3 4c 2f c0 e7 2a 8f a1 2b 22 ad 1b f0 b7 |xg.L/..*..+"....|
000002d0 f0 69 c4 3e b4 c4 c1 e9 00 02 00 00 00 00 00 20 |.i.>........... |
000002e0 7a 7e f7 c6 84 37 0f 97 a5 66 20 22 c5 c7 46 45 |z~...7...f "..FE|
000002f0 89 35 a3 8e d8 7d 62 06 04 55 36 3d 1b ee 24 0c |.5...}b..U6=..$.|
00000300 00 02 00 00 00 00 00 20 0f 41 93 43 87 62 30 07 |....... .A.C.b0.|
00000310 a5 3f c3 a7 f9 ad ed 5b 4d 99 d4 61 b7 50 63 df |.?.....[M..a.Pc.|
00000320 40 7a 8a c8 95 af 0e 56 00 02 00 00 00 00 00 20 |@z.....V....... |
00000330 60 a6 04 8b e8 49 a9 9c 76 ce 8b b2 bb 1d ed 80 |`....I..v.......|
00000340 8c 20 0c 75 98 c4 51 ff 1e 41 af f1 7c fd 23 54 |. .u..Q..A..|.#T|
00000350 00 02 00 00 00 00 00 20 42 58 ce a5 4e 6c 6b 94 |....... BX..Nlk.|
00000360 32 0f 76 4e 89 fa 58 b4 f6 11 a3 27 de 41 ac 29 |2.vN..X....'.A.)|
00000370 99 32 52 c5 2d 6e b0 5e 00 02 00 00 00 00 00 20 |.2R.-n.^....... |
00000380 18 8f e4 e6 69 92 47 a8 3c 58 6b 19 33 36 af 12 |....i.G.<Xk.36..|
00000390 e4 49 4f 2c 27 0c ff 31 8a 57 6d ce 25 8b f5 ca |.IO,'..1.Wm.%...|
000003a0 00 02 00 00 00 00 00 20 65 85 eb 45 67 0d d9 29 |....... e..Eg..)|
000003b0 bb 2f 77 75 65 c3 3b 58 33 cd 63 99 26 a0 36 69 |./wue.;X3.c.&.6i|
000003c0 50 fe c4 55 f7 d1 8f 7f 00 02 00 00 00 00 00 20 |P..U........... |
000003d0 af c0 78 d6 71 ec 5b 7f 36 4c 5a 24 96 65 fa ff |..x.q.[.6LZ$.e..|
000003e0 37 d4 fc c1 cb fe 1a e8 94 8a 91 fe ac d5 68 45 |7.............hE|
000003f0 00 02 00 00 00 00 00 20 55 03 7c 1b fe c0 a5 1c |....... U.|.....|
00000400 09 7f a2 4a 0e fa 49 d1 1f 13 47 d9 07 ba fe 7b |...J..I...G....{|
00000410 86 55 09 db 52 ca f7 40 00 01 2f 07 00 00 00 20 |.U..R..@../.... |
00000420 39 62 11 d5 4c 3e e4 df d7 18 4d 9d e9 20 5b bc |9b..L>....M.. [.|
00000430 c5 04 d7 ad 31 48 51 db a9 da 42 18 2d a1 fe af |....1HQ...B.-...|Thanks On Dec 7, 2015 10:29 PM, "Jakob Borg" notifications@github.com wrote:
Yep, here:
https://nym.se/t/index-v0.11.0.db.rar
This is what I see, and I guess if that is what is actually on disk it's just going to be unclear how the zeroes got there:
Key: 00000000 00 4c 6a 75 64 62 c3 b6 63 6b 65 72 00 00 00 00 |.Ljudb..cker....| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000040 00 60 14 db fc f4 22 e6 dc d2 ee 6b 9d 4f 58 f1 |.`...."....k.OX.| 00000050 c8 ae c7 bb ed bb 4a 7e 1b 64 b5 31 cb 6e 74 e0 |......J~.d.1.nt.| 00000060 02 41 74 6c 61 73 20 53 68 72 75 67 67 65 64 20 |.Atlas Shrugged | 00000070 2d 41 79 6e 20 52 61 6e 64 2f 30 39 33 34 20 2d |-Ayn Rand/0934 -| 00000080 33 39 74 2e 6d 70 33 |39t.mp3|
Value: 00000000 00 00 00 26 41 74 6c 61 73 20 53 68 72 75 67 67 |...&Atlas Shrugg| 00000010 65 64 20 2d 41 79 6e 20 52 61 6e 64 2f 30 39 33 |ed -Ayn Rand/093| 00000020 34 20 2d 33 39 74 2e 6d 70 33 00 00 00 00 01 a4 |4 -39t.mp3......| 00000030 00 00 00 00 56 07 01 65 00 00 00 01 07 19 0a e0 |....V..e........| 00000040 af d3 c2 f9 00 00 00 00 00 00 00 01 00 00 00 00 |................| 00000050 00 00 85 f9 00 00 00 19 00 02 00 00 00 00 00 20 |............... | 00000060 38 c2 72 0c a5 15 90 72 bc 2e 04 ee d6 f0 3f 0d |8.r....r......?.| 00000070 84 b5 2b a6 18 f0 90 dd f3 88 18 13 16 0b 4f 20 |..+...........O | 00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000000a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000000e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000100 b4 af 4d e6 75 a9 b5 9b 61 c4 1a 12 1c 6b 16 28 |..M.u...a....k.(| 00000110 6a 71 77 f5 3c 19 65 be 4c a8 54 ca 93 40 93 75 |jqw.<.e.L.T..@.u| 00000120 00 02 00 00 00 00 00 20 9e 06 69 0e 82 c8 57 0e |....... ..i...W.| 00000130 f9 be b4 57 f1 82 f6 32 4d f1 a5 eb b3 38 a8 21 |...W...2M....8.!| 00000140 ad 71 ce b1 a6 38 24 06 00 02 00 00 00 00 00 20 |.q...8$........ | 00000150 e1 0e be 92 02 0b 30 25 67 e8 b4 00 dd ba 08 b4 |......0%g.......| 00000160 ae f4 88 da f3 e5 96 99 1a 2b db 0d c2 6d bb 08 |.........+...m..| 00000170 00 02 00 00 00 00 00 20 34 d5 b2 54 80 f1 32 55 |....... 4..T..2U| 00000180 eb bc 21 08 ad ab 96 8c 47 2c ca ee 55 b4 88 06 |..!.....G,..U...| 00000190 76 d0 ff ce 28 a4 1f ac 00 02 00 00 00 00 00 20 |v...(.......... | 000001a0 06 f0 a6 9f c0 2f 8f a6 71 89 00 27 30 b2 fe ad |...../..q..'0...| 000001b0 54 19 ed 97 dc 00 18 51 ac 9c 44 35 95 ab d8 30 |T......Q..D5...0| 000001c0 00 02 00 00 00 00 00 20 7a 59 f9 86 4e d4 44 03 |....... zY..N.D.| 000001d0 c7 df fc dc a4 67 65 be cc ec 95 c3 3e 94 46 25 |.....ge.....>.F%| 000001e0 57 e1 59 ed a5 83 58 be 00 02 00 00 00 00 00 20 |W.Y...X........ | 000001f0 20 ce 37 f3 9c 06 f2 88 7a 89 74 58 82 9d ed 02 | .7.....z.tX....| 00000200 1e 30 e6 9d 41 bd 0d 3f ac 8e ed 83 76 93 17 90 |.0..A..?....v...| 00000210 00 02 00 00 00 00 00 20 26 1c 56 16 dc 6f 6d 0e |....... &.V..om.| 00000220 d1 b1 ca 56 4c 14 8c 33 5b bf fa f2 00 e5 33 38 |...VL..3[.....38| 00000230 38 5c 12 e1 86 7d 3a a6 00 02 00 00 00 00 00 20 |8...}:........ | 00000240 25 28 98 d3 4e 30 84 40 ef 06 38 74 d4 49 85 52 |%(..N0.@..8t.I.R| 00000250 a7 7d 00 ac 84 f2 4d e7 57 ed 01 1d c7 dc 8d 8a |.}....M.W.......| 00000260 00 02 00 00 00 00 00 20 07 f2 c2 7f 79 1d 25 09 |....... ....y.%.| 00000270 8b 24 a2 c4 1a 09 e8 ff 7a d4 d6 99 6c cd b0 10 |.$......z...l...| 00000280 96 07 f0 59 f7 66 ce 5c 00 02 00 00 00 00 00 20 |...Y.f........ | 00000290 75 b5 9b 8d 21 fc 23 94 b0 6e 0c a3 be 3b 11 cd |u...!.#..n...;..| 000002a0 d7 66 91 b5 d5 7f 76 75 b5 e0 d1 a7 21 7f 40 93 |.f....vu....!.@.| 000002b0 00 02 00 00 00 00 00 20 1e e1 f6 d5 5c 30 9e af |....... ....\0..| 000002c0 78 67 a3 4c 2f c0 e7 2a 8f a1 2b 22 ad 1b f0 b7 |xg.L/..*..+"....| 000002d0 f0 69 c4 3e b4 c4 c1 e9 00 02 00 00 00 00 00 20 |.i.>........... | 000002e0 7a 7e f7 c6 84 37 0f 97 a5 66 20 22 c5 c7 46 45 |z~...7...f "..FE| 000002f0 89 35 a3 8e d8 7d 62 06 04 55 36 3d 1b ee 24 0c |.5...}b..U6=..$.| 00000300 00 02 00 00 00 00 00 20 0f 41 93 43 87 62 30 07 |....... .A.C.b0.| 00000310 a5 3f c3 a7 f9 ad ed 5b 4d 99 d4 61 b7 50 63 df |.?.....[M..a.Pc.| 00000320 40 7a 8a c8 95 af 0e 56 00 02 00 00 00 00 00 20 |@z.....V....... | 00000330 60 a6 04 8b e8 49 a9 9c 76 ce 8b b2 bb 1d ed 80 |`....I..v.......| 00000340 8c 20 0c 75 98 c4 51 ff 1e 41 af f1 7c fd 23 54 |. .u..Q..A..|.#T| 00000350 00 02 00 00 00 00 00 20 42 58 ce a5 4e 6c 6b 94 |....... BX..Nlk.| 00000360 32 0f 76 4e 89 fa 58 b4 f6 11 a3 27 de 41 ac 29 |2.vN..X....'.A.)| 00000370 99 32 52 c5 2d 6e b0 5e 00 02 00 00 00 00 00 20 |.2R.-n.^....... | 00000380 18 8f e4 e6 69 92 47 a8 3c 58 6b 19 33 36 af 12 |....i.G.<Xk.36..| 00000390 e4 49 4f 2c 27 0c ff 31 8a 57 6d ce 25 8b f5 ca |.IO,'..1.Wm.%...| 000003a0 00 02 00 00 00 00 00 20 65 85 eb 45 67 0d d9 29 |....... e..Eg..)| 000003b0 bb 2f 77 75 65 c3 3b 58 33 cd 63 99 26 a0 36 69 |./wue.;X3.c.&.6i| 000003c0 50 fe c4 55 f7 d1 8f 7f 00 02 00 00 00 00 00 20 |P..U........... | 000003d0 af c0 78 d6 71 ec 5b 7f 36 4c 5a 24 96 65 fa ff |..x.q.[.6LZ$.e..| 000003e0 37 d4 fc c1 cb fe 1a e8 94 8a 91 fe ac d5 68 45 |7.............hE| 000003f0 00 02 00 00 00 00 00 20 55 03 7c 1b fe c0 a5 1c |....... U.|.....| 00000400 09 7f a2 4a 0e fa 49 d1 1f 13 47 d9 07 ba fe 7b |...J..I...G....{| 00000410 86 55 09 db 52 ca f7 40 00 01 2f 07 00 00 00 20 |.U..R..@../.... | 00000420 39 62 11 d5 4c 3e e4 df d7 18 4d 9d e9 20 5b bc |9b..L>....M.. [.| 00000430 c5 04 d7 ad 31 48 51 db a9 da 42 18 2d a1 fe af |....1HQ...B.-...|
— Reply to this email directly or view it on GitHub https://github.com/syndtr/goleveldb/issues/128#issuecomment-162557804.
syndtr
commented
8 years ago Yep, the zeroes are on disk alright. So how hard is it to reproduce this? If you could reproduce this from scratch with issue128 branch that would be great (note that I add checksum on each key/value pair so the on-disk format are incompatible with the master branch). Thanks!
calmh
commented
8 years ago Only got this as reports from a couple of users so far, so no idea on the reproduction unfortunate. Haven't ever seen it myself, and people manage to corrupt their databases all the time for whatever reasons, but this was a different and kind of specific pattern... I'll add some checks where possible to see if we can have this narrowed down somewhat. In the meantime I don't think there's anything we can ask of you. :)
We've seen some weird occurrances of corruption lately that may well be our own bug but we haven't nailed it down. The pattern seems to be that 128 bytes into the data, 128 bytes are overwritten by zeroes. That is, offset 0x80 through 0xff in the dump below shouldn't have been zeroes but other data that would have made sense:
That is as returned from Get(key, nil). I'm fairly sure we generated the correct data (it would be a very odd bug otherwise) and hopefully wrote it to the database, but then I'm not sure what might have happened... Any ideas? Or ideas to the contrary, i.e. it makes absolutely no sense that a block of 128 bytes specifically would have been overwritten so I should go bark up another tree? But if for example the compression writes stuff in blocks of 128 bytes and may silently fail to unpack a block in case there is on disk corruption, or something...
Also, issue #128 \o/