Trying to decode the LZ4 compression (Invalid magic number?)

[kevinvdburgt]

As i am messing around, building a web based map editor, i have some issues decoding the LZ4 compression.

As the data from a map is:

<Buffer 08 00 00 00 01 80 80 40 97 8c 1d ff 09 08 ac 1b 12 90 c0 80 04 0a 07 74 65 72 72 61 69 6e 12 82 c0 80 04 99 39 02 00 ff ff ff ff ff ff ff ff ff ff ff ... >

The first 4 bytes are for the version (https://github.com/Facepunch/Rust.World/blob/master/Assets/Plugins/Rust.World/WorldSerialization.cs#L216) which should be 8 (<Buffer 08 00 00 00>), but after those 4 bytes, it starts with:

<Buffer 01 80 80 40 97 8c 1d ff 09 08 ac 1b 12 90 c0 80 04 0a 07 74 65 72 72 61 69 6e 12 82 c0 80 04 99 39 02 00 ff ff ff ff ff ff ff ff ff ff ff ... >

What should be decoded by LZ4 (https://github.com/Facepunch/Rust.World/blob/master/Assets/Plugins/Rust.World/WorldSerialization.cs#L220). However, when i checkout the documentation for LZ4 the frame should start with a magic number 0x184D2204 (see: https://github.com/lz4/lz4/wiki/lz4_Frame_format.md#general-structure-of-lz4-frame-format).

So, how can i read the map files?

---

Example

As i am writing this in the backend of a NodeJS application, this is how i try to decode it now:

import fs from 'fs';
import lz4 from 'lz4';

const data = fs.readFileSync('./map/proceduralmap.3500.26341726.169.map');
const map = data.slice(4); // Slice versioning number
const decoded = lz4.decode(map);
console.log(decoded);

// id magic number: 40808001 @0
//     at Decoder.emit_Error (/Users/kevin/dev/test/node-rust-map/node_modules/lz4/lib/decoder_stream.js:64:22)
//     at Decoder.read_MagicNumber (/Users/kevin/dev/test/node-rust-map/node_modules/lz4/lib/decoder_stream.js:93:8)
//     at Decoder._main (/Users/kevin/dev/test/node-rust-map/node_modules/lz4/lib/decoder_stream.js:289:25)
//     at Decoder._transform (/Users/kevin/dev/test/node-rust-map/node_modules/lz4/lib/decoder_stream.js:60:7)
//     at Decoder.Transform._read (_stream_transform.js:185:10)
//     at Decoder.Transform._write (_stream_transform.js:173:12)
//     at doWrite (_stream_writable.js:407:12)
//     at writeOrBuffer (_stream_writable.js:393:5)
//     at Decoder.Writable.write (_stream_writable.js:290:11)
//     at Decoder.Writable.end (_stream_writable.js:573:10)

[kevinvdburgt]

Figured it out, Facepunch is using this library, which is not fully compatible with the original LZ4 (https://github.com/MiloszKrajewski/lz4net#compatibility).

Currently learning more about how LZ4 works, so i can make the decoder in Node and continue on the webbased map editor :)

[realkarmakun]

@kevinvdburgt Have you figured it out? I'm trying to decode and serialize it from java, and writing JNI for and old lib doesn't sound applealing