npm package appears corrupt

[saimonmoore]

Unable to install this package:

 yarn add --registry https://registry.npmjs.org/ kafkajs-lz4
yarn add v1.22.4
[1/5] 🔍  Validating package.json...
[2/5] 🔍  Resolving packages...
[3/5] 🚚  Fetching packages...
error https://registry.npmjs.org/lz4/-/lz4-0.6.3.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, link '/Users/saimon.moore/Library/Caches/Yarn/v6/npm-lz4-0.6.3-78df6bb69a36d7db6c2e849494876ba6e38e66d6-integrity/node_modules/lz4/build/Release/obj.target/build/Release/lz4.node' -> '/Users/saimon.moore/Library/Caches/Yarn/v6/npm-lz4-0.6.3-78df6bb69a36d7db6c2e849494876ba6e38e66d6-integrity/node_modules/lz4/build/Release/obj.target/lz4.node'"
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

[paambaati]

This seems to be an upstream issue — I can replicate this issue when I try to install the lz4 package.

$ yarn add lz4

yarn add v1.22.4
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
error https://registry.yarnpkg.com/lz4/-/lz4-0.6.3.tgz: Extracting tar content of undefined failed, the file appears to be corrupt: "ENOENT: no such file or directory, link '/Users/gp/Library/Caches/Yarn/v6/npm-lz4-0.6.3-78df6bb69a36d7db6c2e849494876ba6e38e66d6-integrity/node_modules/lz4/build/Release/obj.target/build/Release/lz4.node' -> '/Users/gp/Library/Caches/Yarn/v6/npm-lz4-0.6.3-78df6bb69a36d7db6c2e849494876ba6e38e66d6-integrity/node_modules/lz4/build/Release/obj.target/lz4.node'"

@saimonmoore Please open an issue in the lz4 project's issue tracker.

[saimonmoore]

Was already opened: https://github.com/pierrec/node-lz4/issues/97

[leoparis89]

It's due to a breaking change in yarn v1.22. Running yarn policies set-version 1.21 fixed the problem for me.

[paambaati]

@saimonmoore Can you confirm the the workaround posted above by @leoparis89 fixes this issue?

yarn policies set-version 1.21

[paambaati]

Can you folks try the latest v2 beta release? See https://github.com/indix/kafkajs-lz4/pull/7#issuecomment-738652072

yarn add kafkajs-lz4@2.0.0-beta.0

[link2malkeet]

Hey paambati We are getting a typeerror, we tried both version 1.2.1 and the latest beta - 2.0.0-beta: TypeError: kafkajs_lz4_1.default is not a constructor Thanks

[paambaati]

@link2malkeet Can you share the version of Node and the relevant piece of code where you’re using this package?

[link2malkeet]

Thanks for coming so quickly, That's resolved, actually we followed the documentation and found the issue with the way we need to call codec on the instance of LZ4Codec with this code snippet, we manage to create an instance of LZ4Codec const LZ4Codec = require('kafkajs-lz4'); // import LZ4Codec fromkafkajs-lz4did not work. const LZ4Instance = new LZ4Codec(); CompressionCodecs[CompressionTypes.LZ4] = LZ4Instance.codec; But After this configuration, still we are getting the same old error saying LZ4 compression not implemented.

Usage looks very simple but could not get it working yet. 😓 Thanks Mel

[paambaati]

@link2malkeet This should work though, I'm not sure of the root-cause of your issue.

Can you follow the examples (or even our tests); they're fairly straightforward and are known to work.

If that doesn't work, I'm afraid we'll need to know more about your setup and your code.

[link2malkeet]

Well our setup is pretty straight forward. Before we create Producer and Consumer instances, we simply add the compression configuration as suggested so build and deployment works fine but when consumer listens to the kafka topic(which has messages compressed with LZ4), the subscription throws this error saying LZ4 compression not implemented Node: v14.15.0 Yarn: v1.21.1

can you point out where we are supposed to add LZ4 configuration?

[paambaati]

where we are supposed to add LZ4 configuration?

@link2malkeet They're supposed to be added to the producer.send method. You can check our tests (see https://github.com/indix/kafkajs-lz4/blob/5efecce6fc8e746d43bc21298e169eb62eb65b5a/test/index.test.ts#L59-L112) and the official documentation (see https://kafka.js.org/docs/producing).

[link2malkeet]

Finally it worked. Instead of using this library, we simple took the index.ts file source.

Seems this is culprit there are 2 things exporting LZ4Codec export default class LZ4Codec vs module.exports = LZ4Codec;

[link2malkeet]

Can you guys please update it with the change so that we could use this libraray with the required fix?

[paambaati]

@link2malkeet I'm sorry, but I don't think that's the right approach. I'd recommend you read up about CommonJS and ES6 imports and how they're different.

For a proper solution to your issue, like I've said before, I'd need more information — that includes your Node version, TS version (if you're using any), Babel config (if you're using any) and more relevant code (how you're importing this package, how/where you're using the produce & consume methods, etc.)

If you'd like to pursue this, I'd recommend you open a new issue as yours seems to be different from this.

[yayapao]

Same issues when i use nestjs monorepo. I fininally solve it by use const LZ4Codec = require('kafkajs-lz4'), and here is my tsconfig.json:

{
  "compilerOptions": {
    "module": "CommonJS",
    "target": "ES2017"
    "Other configs": "..."
  }
}