nvite / waoffle

Redis AOF parser
ISC License
51 stars 12 forks source link

rwaoffle producing invalid aof #4

Open yocontra opened 6 years ago

yocontra commented 6 years ago
$ redis-check-aof tile38/appendonly.aof
0x          abf2aa: Expected \r\n, got: 7d0d

AOF was generated using rwaoffle, redis-check-aof fails on it though. The file is 2GB so lmk if you need it and we can arrange it.

yocontra commented 6 years ago

Interesting, but it seems like changing the line endings on the file from \n to \r\n changes the output AOF. Didn't make a difference though, same issue:

redis-check-aof tile38/appendonly.aof
0x          ac12cb: Expected \r\n, got: 7d0d
AOF analyzed: size=1876544426, ok_up_to=11273597, diff=1865270829
AOF is not valid. Use the --fix option to try fixing it.
yocontra commented 6 years ago

Chunk starting from byte 11273597 (reported as the beginning of corruption):

$ tail -c +11273597 tile38/appendonly.aof

*6
$3
set
$10
boundaries
$3
blm
$6
object
$3341
{"type":"MultiPolygon","coordinates":[],"properties":{"id":"blm","type":"country","name":"Saint
$26
Barthélemy","code":"BLM"}}
*5
$3
set
$10
boundaries
$3
mys
$6
object
$584783
{"type":"MultiPolygon","coordinates":[],"properties":{"id":"mys","type":"country","name":"Malaysia","code":"MYS"}}
*6
$3
set
$10
boundaries
$3
sle
$6
object
$175529
{"type":"MultiPolygon","coordinates":[]}

Only included the 3 commands from that chunk, as there was ~84K total in the file. Seems interesting the control character in the middle of the JSON in one, but not in the other.

yocontra commented 6 years ago

Ah - found the issue, rwaoffle splits by \s to determine the parts, so any values that have whitespace in them get bungled. https://github.com/nvite/waoffle/blob/master/lib/generate.js#L40