search

SpringMT / zstd-ruby

Ruby binding for zstd(Zstandard - Fast real-time compression algorithm)
https://github.com/facebook/zstd
BSD 3-Clause "New" or "Revised" License
69 stars 16 forks source link

symbol lookup error after upgrade to 1.5.1 #37

Closed jarthod closed 2 years ago

jarthod commented 2 years ago

Hello :wave:

I just tried upgrading from 1.5.0.0 to 1.5.1.0 but I'm getting this exception: symbol lookup error: /home/adrien/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/zstd-ruby-1.5.1.0/lib/zstd-ruby/zstdruby.so: undefined symbol: HUF_decompress4X1_usingDTable_internal_bmi2_asm_loop which is causing the ruby program to exit with status 127. (the gem installation and native extension compilation goes without problem).

I have the same problem on two different machines running Ubuntu 21.10 and 21.04 (if that's of any relevance). I'm using ruby 2.6.4 and I'm using zstd-ruby only for the mongo driver (version 2.17.0). This is the commit showing how the mongo gem is using zstd-ruby: https://github.com/mongodb/mongo-ruby-driver/commit/c07aae3a564f464db450deeb0f2f43c4d0984caa, nothing very special here IMO, looks like it's just basic Zstd.decompress and Zstd.compress.

I can't reproduce with a simple console test:

> Zstd.compress('test')
=> "(\xB5/\xFD \x04!\x00\x00test"
> Zstd.decompress(Zstd.compress('test'))
=> "test"

But I can reproduce reliably with any Mongo query:

> User.first
rails_console: symbol lookup error: /home/adrien/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/zstd-ruby-1.5.1.0/lib/zstd-ruby/zstdruby.so: undefined symbol: HUF_decompress4X1_usingDTable_internal_bmi2_asm_loop

(IRB exits with status 127)

Any idea? Thanks!

SpringMT commented 2 years ago

I try to fix this problem in https://github.com/SpringMT/zstd-ruby/pull/38. Can you test using this branch fix-asm-support? Ex(in Gemfile)

gem "zstd-ruby", branch: "fix-asm-support"
SpringMT commented 2 years ago

I released v1.5.1.1 fixed the error. https://rubygems.org/gems/zstd-ruby/versions/1.5.1.1. Please test it!

jarthod commented 2 years ago

Thanks for the quick fix ! 1.5.1.1 seems to be working fine :) For my understanding what is "ASM support"? why wasn't my simple test failing?

SpringMT commented 2 years ago

Zstd Compression are 4 block types. https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#blocks

I think your simple text compression is Raw_Block. 00 that is 1 byte before test in the result string "(\xB5/\xFD \x04!\x00\x00test" shows Raw_Block.

Raw_Block doesn't use ZSTD_decompressBlock_internal that uses Compressed_Block. https://github.com/facebook/zstd/blob/e34e58b458d6460b7eabdf4bde9d6b7da90a842d/lib/decompress/zstd_decompress.c#L917-L928

ZSTD_decompressBlock_internal -> ZSTD_decodeLiteralsBlock -> HUF_decompress4X_usingDTable_bmi2 https://github.com/facebook/zstd/blob/d5933a40096605d13b9a21269c6c34d06d66013b/lib/decompress/zstd_decompress_block.c https://github.com/facebook/zstd/blob/d5933a40096605d13b9a21269c6c34d06d66013b/lib/decompress/zstd_decompress_block.c#L177-L204 https://github.com/facebook/zstd/blob/252ef866fb0f018ea0ff596c49027201442d7b00/lib/decompress/huf_decompress.c

HUF_decompress4X_usingDTable_bmi2 -> HUF_decompress4X1_usingDTable_internal

In HUF_decompress4X1_usingDTable_internal, ZSTD_ENABLE_ASM_X86_64_BMI2 macro is used.

https://github.com/facebook/zstd/blob/859e0500ab98e07c680b270efa0f2dcdf11391cf/lib/common/portability_macros.h#L122-L129

This macro is for ASM macros . https://github.com/facebook/zstd/issues/2789 https://github.com/facebook/zstd/pull/2722 https://github.com/facebook/zstd/pull/2893

The x86-64 Huffman assembly implementation is .S files.

When 1.5.1.0 release, .S files didn't include src, and maybe your environment matches ZSTD_ENABLE_ASM_X86_64_BMI2, so HUF_decompress4X1_usingDTable_internal -> HUF_decompress4X1_usingDTable_internal_bmi2_asm -> HUF_decompress4X1_usingDTable_internal_bmi2_asm_loop then the error happened.

jarthod commented 2 years ago

I see, thanks for the explanation!