indix / kafkajs-lz4

🗜 LZ4 compression codec for KafkaJS
MIT License
15 stars 8 forks source link

Does not work on node 18 #16

Open mark-b-ab opened 1 year ago

mark-b-ab commented 1 year ago

Explanation here https://github.com/ukyo/lz4.js/issues/14#issue-1434765066

Error for Google index: TypeError: Failed to parse URL from /Users/mark/Phpstorm/dtq/node_modules/lz4-asm/dist/_lz4.wasm

mark-b-ab commented 1 year ago

One more workaround

//Explanation: https://github.com/ukyo/lz4.js/issues/14
const originalFetch = global.fetch;
global.fetch = undefined;
const LZ4 = require('kafkajs-lz4');
global.fetch = originalFetch;

CompressionCodecs[CompressionTypes.LZ4] = new LZ4().codec;
eau-de-la-seine commented 1 year ago

Hi, this workaround doesn't work for me and I still get the same error, I think it's because I use esbuild, all the require() are called first!

I don't know if the authors still work on this project (last commit was +2 years ago), but yeah the lib is broken for Nodejs18

mark-b-ab commented 1 year ago

You can try to write your own wrapper around other lz4 library or switch to snappy as we did

eau-de-la-seine commented 1 year ago

Oh hi @mark-b-ab, thanks! I don't like workarounds and I was not going back to NodeJSv16 because an old lib will probably never get updated, so I have switched to Snappy too even if I have hesitated with the ZSTD!

NB: Despite having compression.type=producer by default at broker-side, I haven't noticed any compression optimization by Snappy with the kafkajs client though (the last commit on the Snappy lib is +4y)

jimmywarting commented 10 months ago

bump

athira009 commented 8 months ago

Hi all, I am getting the same error for all node versions of 18 or higher. Also, I cannot able to use snappy. Is there any other alternative for this?

Marko298 commented 8 months ago

Hi all, I am getting the same error for all node versions of 18 or higher. Also, I cannot able to use snappy. Is there any other alternative for this?

Use workarounds. Works fine, running in prod, all good

athira009 commented 7 months ago

Hi @Marko298 , thanks for the quick reply. Could you specify the workarounds?

Marko298 commented 7 months ago

Hi @Marko298 , thanks for the quick reply. Could you specify the workarounds?

https://github.com/indix/kafkajs-lz4/issues/16#issuecomment-1302235991

athira009 commented 7 months ago

That didn't work @Marko298

simyara commented 1 month ago

bump