search

manzt / numcodecs.js

Buffer compression and transformation codecs
MIT License
29 stars 6 forks source link

Add lz4 codec #16

Closed manzt closed 3 years ago

manzt commented 4 years ago

Hmm there is a bug. Although tests are passing, codec.encode isn't compressing the array... and I can't decode a zarr-python generated chunk.

# create python fixture
import numpy as np
import zarr
z = zarr.open('data.zarr', shape=(100, 100), dtype='i4', compressor=LZ4())
z[:] = np.arange(100 * 100).reshape(100, 100).astype('i4')
// test decompression in node
import fs from 'fs';
import LZ4 from '../dist/lz4.mjs';

const data = fs.readFileSync('data.zarr/0.0');
const bytes = new Uint8Array(data);
const codec = new LZ4();

(async _ => {
  console.log(bytes)
  const decoded = await codec.decode(bytes);
  console.log(decoded);
})();
$ node index.mjs
Uint8Array(40035) [
  128, 56,  1, 0, 25,  0,  1, 0, 35, 240, 63, 15,
    0, 18, 64, 8,  0, 19,  8, 8,  0,  19, 16,  8,
    0, 19, 20, 8,  0, 19, 24, 8,  0,  19, 28,  8,
    0, 19, 32, 8,  0, 19, 34, 8,  0,  19, 36,  8,
    0, 19, 38, 8,  0, 19, 40, 8,  0,  19, 42,  8,
    0, 19, 44, 8,  0, 19, 46, 8,  0,  19, 48,  8,
    0, 19, 49, 8,  0, 19, 50, 8,  0,  19, 51,  8,
    0, 19, 52, 8,  0, 19, 53, 8,  0,  19, 54,  8,
    0, 19, 55, 8,
  ... 39935 more items
]
(node:47294) UnhandledPromiseRejectionWarning: RangeError: Invalid typed array length: 4294967284
    at new Uint8Array (<anonymous>)
    at Object.e (file:///Users/trevormanz/GitHub/manzt/numcodecs.js/dist/lz4.mjs:37:74)
    at q (file:///Users/trevormanz/GitHub/manzt/numcodecs.js/dist/lz4.mjs:42:108)
    at <anonymous>:wasm-function[59]:0x100a
    at <anonymous>:wasm-function[69]:0x16d9
    at <anonymous>:wasm-function[111]:0x3668
    at w (<anonymous>:wasm-function[121]:0x3a5c)
    at d.dynCall_iii (file:///Users/trevormanz/GitHub/manzt/numcodecs.js/dist/lz4.mjs:47:57)
    at dynCall_iii_1 (eval at Xa (file:///Users/trevormanz/GitHub/manzt/numcodecs.js/dist/lz4.mjs:28:235), <anonymous>:4:12)
    at Object.decompress (eval at Sa (file:///Users/trevormanz/GitHub/manzt/numcodecs.js/dist/lz4.mjs:25:449), <anonymous>:8:10)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:47294) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:47294) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.