Replace sse4_crc32 with @node-rs/crc32

[Brooooooklyn]

https://github.com/Brooooooklyn/node-rs/tree/master/packages/crc32

Here is benchmark on my machine:

Model Name: MacBook Pro
Model Identifier: MacBookPro15,1
Processor Name: Intel Core i7
Processor Speed: 2.6 GHz
Number of Processors: 1
Total Number of Cores: 6
L2 Cache (per Core): 256 KB
L3 Cache: 12 MB
Hyper-Threading Technology: Enabled
Memory: 16 GB

Benchmark:

@node-rs/crc32c for inputs 1024B x 5,264,095 ops/sec ±1.20% (93 runs sampled)
@node-rs/crc32c for inputs 16829745B, avg 2054B x 263 ops/sec ±11.01% (71 runs sampled)
sse4_crc32c_hw for inputs 1024B x 3,605,658 ops/sec ±1.00% (91 runs sampled)
sse4_crc32c_hw for inputs 16829745B, avg 2054B x 211 ops/sec ±1.08% (72 runs sampled)
sse4_crc32c_sw for inputs 1024B x 1,533,277 ops/sec ±0.83% (92 runs sampled)
sse4_crc32c_sw for inputs 16829745B, avg 2054B x 99.14 ops/sec ±1.90% (73 runs sampled)
js_crc32c for inputs 1024B x 439,854 ops/sec ±2.55% (89 runs sampled)
js_crc32c for inputs 16829745B, avg 2054B x 26.27 ops/sec ±2.87% (47 runs sampled)
js_crc32 for inputs 1024B x 422,623 ops/sec ±2.52% (83 runs sampled)
js_crc32 for inputs 16829745B, avg 2054B x 28.34 ops/sec ±1.64% (51 runs sampled)
+-----------------+-------------------+----------------------+
|                 │ 1024B             │ 16829745B, avg 2054B |
+-----------------+-------------------+----------------------+
| @node-rs/crc32c │ 5,264,095 ops/sec │ 263 ops/sec          |
+-----------------+-------------------+----------------------+
| sse4_crc32c_hw  │ 3,605,658 ops/sec │ 211 ops/sec          |
+-----------------+-------------------+----------------------+
| sse4_crc32c_sw  │ 1,533,277 ops/sec │ 99.14 ops/sec        |
+-----------------+-------------------+----------------------+
| js_crc32c       │ 439,854 ops/sec   │ 26.27 ops/sec        |
+-----------------+-------------------+----------------------+
| js_crc32        │ 422,623 ops/sec   │ 28.34 ops/sec        |
+-----------------+-------------------+----------------------+

[ashi009]

This is interesting. By implementing CRC32C in rust alone can make such a performance difference?

[Brooooooklyn]

Seems like not only the differences between the languages, but also the implementation inside.

[kiyui]

Bumping this as sse4_crc32 is not working on darwin-arm64 at the moment 😅

> require('@node-rs/crc32')
{ crc32: [Function: crc32], crc32c: [Function: crc32c] }
> require('sse4_crc32')
dyld: lazy symbol binding failed: Symbol not found: __Z18sse42_is_availablev
  Referenced from: /Users/dafne/Workspaces/Piktochart/piktostory/node_modules/sse4_crc32/build/Release/crc32c.node
  Expected in: flat namespace

dyld: Symbol not found: __Z18sse42_is_availablev
  Referenced from: /Users/dafne/Workspaces/Piktochart/piktostory/node_modules/sse4_crc32/build/Release/crc32c.node
  Expected in: flat namespace

[Brooooooklyn]

@kiyui this PR is staled.. Run yarn add @node-rs/crc32 to install latest version and it support darwin-arm64 natively

[kiyui]

The @google-cloud/storage docs recommend installing fast-crc32c to speed up crc32c computation. In turn, @google-cloud/storage requires hash-stream-validation which in turn requires this library.

I would not be bumping this issue if not for this coupling.

[abhiaiyer91]

Hi, just revisiting this issue. Since google node-storage depends on this module can we get the upstream changes of @chainsafe/fast-crc32c and published?

[johngrogg]

Bumping this ask. We're using fast-crc32c in our monorepo, and until this is resolved the entire repo fails to run on M1 macs since the current sse4_crc32 dependency causes a node crash during package import.

[madebymichaelg]

Hey y'all, also wanted to jump in and mention this is a blocker in my organization. If we could merge https://github.com/ashi009/node-fast-crc32c/pull/28 that would be great!

[jimmcslim]

Hello everybody! If you have landed here because you are trying to use @google-cloud/storage on an Apple Silicon Mac, I have raised an issue on the GitHub repo for this library... linked above. Hopefully Google updates their library to remove this dependency!