Both of these copy the input dictionary buffer, allowing the caller to release them. My issue with this is that I am storing dictionaries used for both compression and decompression in the same programs. Compressing data written to storage, and decompressing from storage.
We have a bunch of tenants, thus a bunch of dictionaries. With the current state, this means having two copies of each dictionary.
If I understand correctly, wrapping the *_byReference functions could share the same backing dictionary with both compressors and decompressors. This requires a bit more cooperation from the client program, but this can be noted in the documentation that it's an optimization path.
coxley
commented
2 months ago
Problem
The current library wraps these two functions from zstd:
Both of these copy the input dictionary buffer, allowing the caller to release them. My issue with this is that I am storing dictionaries used for both compression and decompression in the same programs. Compressing data written to storage, and decompressing from storage.
We have a bunch of tenants, thus a bunch of dictionaries. With the current state, this means having two copies of each dictionary.
If I understand correctly, wrapping the
*_byReferencefunctions could share the same backing dictionary with both compressors and decompressors. This requires a bit more cooperation from the client program, but this can be noted in the documentation that it's an optimization path.What are your thoughts, @valyala?