search

SpringMT / zstd-ruby

Ruby binding for zstd(Zstandard - Fast real-time compression algorithm)
https://github.com/facebook/zstd
BSD 3-Clause "New" or "Revised" License
69 stars 16 forks source link

Add skippable frame support #55

Closed lowderdev closed 1 year ago

lowderdev commented 1 year ago

My team uses zstd skippable frames to embedded metadata about files and it doesn't look like this gem currently supports that feature of zstd.

Are there any plans to support that feature? Is this a contribution a new comer could potentially work on?

SpringMT commented 1 year ago

Do I add an option like below(skippable_frame_text option) and add the Skippbale Frame implementation?

compressed_data = Zstd.compress(data, compression_level, skippable_frame_text: "test")
SpringMT commented 1 year ago

I will use fixed Skippable frame magic number(0x184D2A50) , is that a problem?

lowderdev commented 1 year ago

I think we need something more like:

frame  = Zstd.skippable_frame(data)

We have a data structure like this:

File:
  [header (skippable frame)][data (zstd compressed)]

Archive:
  [archive header (skippable frame)][File][File][...][File][archive trailer (skibbable frame)]

So I think we would need to be able to create skippable frames on their own and write them to a file, then append other zstd frames to the file. I'm not sure how this would interact with the current implementation of the decompress method.

That Skippable frame magic number(0x184D2A50) is fine, and is what we are using to do the same thing in Kotlin.

SpringMT commented 1 year ago

I create the PR https://github.com/SpringMT/zstd-ruby/pull/56 @lowderdev Can you test using this branch feature/add-skippable_frame ?

Zstd.read_skippable_frame(compressed_data_with_skippable_frame)

See below for detailed usage. https://github.com/SpringMT/zstd-ruby/pull/56/files#diff-704d8a2c165fe18547c1ab40efb732d5d825c5c1ff3b18c3c0ab62cec3ee06a6

lowderdev commented 1 year ago

Hi @SpringMT. Let me try it out and get back to you!

SpringMT commented 1 year ago

@lowderdev How about testing? I am planning to merge and release this PR by 2023/03/25.

lowderdev commented 1 year ago

My apologies @SpringMT! This got away from me.

I just ran your branch on a zstd compressed file created by a service at my place of work and it worked!

data = File.open('spec/single-txt-file-archive').read
Zstd.read_skippable_frame(data)
=> "LGKARCH!\x01\x00\x00\x00\x00\x00\x00\x00\x93\x03\x13\xAA\x02\x05J\xD0\x93Q\xCDZc\xFD\x81n"

Thanks for responding to this feature request!

SpringMT commented 1 year ago

Release v1.5.4.1 https://rubygems.org/gems/zstd-ruby/versions/1.5.4.1