Open wildug opened 1 year ago
I can see how vectorizing would reduce overhead from python callbacks. Unfortunately, vectorizing is only possible for encoding; when decoding a symbol, the decoder cannot know where to evaluate the ppf before it has decoded all preceding symbols (except in case of the ChainCoder
). I'll have to think a bit what the best API would be to reflect this asymmetry (and to ideally still support vectorization for decoding with a ChainCoder
).
In my usecase I want to compress a large amount of data with a custom entropy model. Unfortunately this takes quite some time since for each compressed symbol the cdf is called. I can't straight up use the scipy model adapter since I'm using a mixture distribution which is not implemented in scipy.
Here's my dummy code:
Is it possible to take care of vectorizable cdfs in the custom model adapter to allow for a speed up?