search

NVIDIA / nvcomp

Repository for nvCOMP docs and examples. nvCOMP is a library for fast lossless compression/decompression on the GPU that can be downloaded from https://developer.nvidia.com/nvcomp.
Other
561 stars 78 forks source link

Maximum data size for compression #94

Open DrivenByNostalgia opened 10 months ago

DrivenByNostalgia commented 10 months ago

I have integrated nvCOMP (version nvcomp_3.0.5_windows_12.x) into our system for the compression of scientific data (GIS, simulation, …) and have noticed that compression fails for data of 2^32 bytes or more, while the compression of a buffer of size 2^32 - 1 works flawlessly.

Specifically, nvcompManagerBase::compress throws the exception "Encountered Cuda Error: 2: 'out of memory'." after calling nvcompManagerBase::configure_compression with a decomp_buffer_size >= 2^32.

If the chosen compression manager is a DeflateManager or GdeflateManager, the call of compress first outputs a warning to std::cerr:

"WARNING: In nvcompBatchedGdeflateCompressGetTempSize: Might not have enough memory available on this GPU. Require 22.002 GB but detected only 0 GB available"

Both of the reported numbers in this case seem wrong to me, but this is not my main issue, as we are not planning to use Deflate/Gdeflate anyways.

Is this an undocumented hard limit for the file size that nvCOMP can handle? I don't mind if there is a limit of 4 GB to the buffer size even with much more available memory. However, in this case, it would be nice to know and query this limit beforehand so to not run into an exception.

This has been tested on a system with:

0 - Device name: NVIDIA GeForce RTX 3060 0 - CUDA Driver Version / Runtime Version: 12.3 / 12.3 0 - CUDA Capability Major/Minor version number: 8.6 0 - Total amount of global memory: 12.00 GBytes

akshaysubr commented 10 months ago

@DrivenByNostalgia Thanks for reporting this. You're right to expect that a buffer sizes >= 2^32 should work or a limitation should be documented. Can you share a minimal repro of this issue?

github-actions[bot] commented 9 months ago

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

DrivenByNostalgia commented 9 months ago

No, I'm sorry, it's not within my time to create a reproducer. But as this is an issue that can easily be circumvented in the calling code, it's not a big issue after all.

github-actions[bot] commented 8 months ago

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

github-actions[bot] commented 5 months ago

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.