Memory allocation routines

[GoogleCodeExporter]

I have my own memory allocator that is used in my code and increase memory 
allocation performance. WebP is very great lib with incredibly complex code. I 
have to use it but I have to merge changes from each new version to my code 
because there are no separated memory allocation routines.
It would be perfect if all memory allocation/freeing operation was made through 
simple set separate function (WebpSafeAlloc/WebpSafeFree etc) but not using 
malloc/free functions. It is ok for me if it would be macro but not function. 
I can do it myself but I have no access to code. 

Original issue reported on code.google.com by SergeySo...@gmail.com on 25 Mar 2014 at 6:08

[GoogleCodeExporter]

good suggestion. The WebPSafeAlloc() functions were introduced for 
input-dependent allocation (that's why most of single-object allocation are 
still using malloc()), but it wouldn't hurt to unify all the memory allocation 
calls.

Original comment by pascal.m...@gmail.com on 25 Mar 2014 at 10:07

[GoogleCodeExporter]

I would suggest to use more allocations on the stack rather than defaulting to 
malloc. Some larger images require over 500 small allocations to decompress 
from within webp.

In our project we are hooking with -Dmalloc=webp_hooked_malloc

On a side note, webp uses an extreme amount of RAM to decompress images when 
not using incremental mode. To decompress a lossy 2048x2048 RGBA image, webp 
requires about 16.5 megabytes of allocations. Including the output buffer, you 
need around 32.5 megabytes on dynamically allocated memory.

Original comment by yo...@phobicgames.com on 26 Mar 2014 at 12:51

[GoogleCodeExporter]

You can have a look at this patch (still being tested):

https://gerrit.chromium.org/gerrit/69362

> I would suggest to use more allocations on the stack rather than defaulting 
to malloc. Some larger images require over 500 small allocations to decompress 
from within webp.

I guess you are referring to lossless compression mode, right? Indeed, this one 
has a lot of objects to allocate, in number that are data-dependent. That's why 
static allocation is harder to predict. Plus, we try to limit the stack size to 
8k overall, which means some single objects that could be on stack are using 
malloc() nevertheless.

But thanks to the instrumentation of malloc/free() (in this patch), we'll have 
a better idea of potential memory optimizations to avoid the fragmentation 
you're alluding to.

> On a side note, webp uses an extreme amount of RAM to decompress images when 
not using incremental mode. To decompress a lossy 2048x2048 RGBA image, webp 
requires about 16.5 megabytes of allocations. Including the output buffer, you 
need around 32.5 megabytes on dynamically allocated memory.

Do you have transparency plane used in this image?
Can you send the webp file (privately if needed) for inspection?

Original comment by s...@google.com on 26 Mar 2014 at 2:23

[GoogleCodeExporter]

https://gerrit.chromium.org/gerrit/#/c/69362/ was merged.

The public API is still not ready for full customizable memory management 
(there's still some gray
area mentioned in the patch's description), but if you're building from sources 
and using the advanced
decoding API you are probably in good shape.

Original comment by pascal.m...@gmail.com on 28 Mar 2014 at 6:24

[GoogleCodeExporter]

following this thread up, note that this patch:

https://gerrit.chromium.org/gerrit/#/c/69891/ (merged)

will reduce the memory traffic for lossless compression. More to come!

Original comment by pascal.m...@gmail.com on 26 Apr 2014 at 8:31

• Changed state: Started