search

NVIDIA / nvcomp

Repository for nvCOMP docs and examples. nvCOMP is a library for fast lossless compression/decompression on the GPU that can be downloaded from https://developer.nvidia.com/nvcomp.
Other
559 stars 78 forks source link

[QST] CPU compression for decompression with high-level interface #86

Open technillogue opened 1 year ago

technillogue commented 1 year ago

How can I generate the metadata needed for decompression?

github-actions[bot] commented 1 year ago

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

technillogue commented 1 year ago

Any update on this? I have a bunch of old data that needs to be compressed and it would be unfortunate to spin up GPUs only for compression

eschmidt-nvidia commented 1 year ago

Hi @technillogue, we're looking at producing a binary that would do this for you (i.e. produce an HLIF buffer using the CPU). Which formats are you interested in?

If you have a significant amount of old data, why not do this on GPU?

technillogue commented 1 year ago

Mostly Gdeflate, though Zstd would be good too. Is the HLIF format proprietary?

I have some terabytes of data that sometimes needs to be loaded very quickly. Compression is cost-sensitive, not time-sensitive.

On Tue, Aug 22, 2023, 8:53 PM eschmidt-nvidia @.***> wrote:

Hi @technillogue https://github.com/technillogue, we're looking at producing a binary that would do this for you (i.e. produce an HLIF buffer using the CPU). Which formats are you interested in?

If you have a significant amount of old data, why not do this on GPU?

— Reply to this email directly, view it on GitHub https://github.com/NVIDIA/nvcomp/issues/86#issuecomment-1688744132, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHG4G2LAEAW5XTBPXQJNNDXWT52XANCNFSM6AAAAAAZMHPKC4 . You are receiving this because you were mentioned.Message ID: @.***>

eschmidt-nvidia commented 1 year ago

I see. If I understand correctly, based on my experiments GPU and CPU will provide similar throughput/$. I've tested this on H100 compared to Genoa / SPR CPUs.

Given that you're not time sensitive, have you investigated the GDeflate high-compression mode? This could provide cost savings if you're storing the data for a long time.

We're looking at adding a similar mode to ZSTD.

The format isn't proprietary but we haven't had time to produce a public document that fully describes it.

technillogue commented 1 year ago

Interesting, I'm not sure what prices you have access to that make that work out but I can give it a shot and do compression a little more efficiently

As of June, GDeflate high-compression mode was broken for HLIF https://github.com/NVIDIA/nvcomp/issues/81#issuecomment-1597533786. I'm mostly interested in compressing model finetunes, not datasets, so almost all gains are from entropy coding and not dictionary compression. When trying the LLIF benchmark entropy-only had about the same compression ratio as high compression. I imagine there's ways to tune specifically entropy coding to be higher compression though.

eschmidt-nvidia commented 1 year ago

I responded to the earlier issue. This should be fixed.

Interesting regarding entropy-only being better. Have you tried our ANS, bitcomp, or Cascaded formats?

github-actions[bot] commented 1 year ago

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

github-actions[bot] commented 10 months ago

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.