search

FLIF-hub / FLIF

Free Lossless Image Format
Other
3.72k stars 229 forks source link

Massive memory consumption #33

Closed bobobo1618 closed 9 years ago

bobobo1618 commented 9 years ago

Attempting to convert a large (24MP) 16-bit PNG results in FLIF eating a ton (>40GB) of memory.

This memory appears to be primarily produced in create_transform (heap profile produced by compiling FLIF with debug enabled and gperftools linked):

screen shot 2015-10-03 at 21 47 40
bobobo1618 commented 9 years ago

Here's a profile with more extreme memory usage (largely the same graph still):

screen shot 2015-10-03 at 21 50 00
bobobo1618 commented 9 years ago

Looking into the code, I saw that this is mainly used for FLIF decoding while the command I'm running (flif <input>.png <output>.flif) is intended to encode. When I converted the image to PGM before running flif, it worked fine and memory usage was reasonable.

I suspect this is an issue resulting from mistakenly trying to decode a PNG as a flif.

FLIF-hub commented 9 years ago

I wonder what is happening here. Perhaps you are not using the libpng png decoder but Stb ? (e.g. I think the Windows build uses that one). It does not support 16-bit PNG, so maybe something crazy is happening.

bobobo1618 commented 9 years ago

No, I'm definitely using libpng, I'm on Mac (clang) and it's in the CXXFLAGS. I get the issue when attempting to encode from PPM, PAM and PNG. Encoding from PGM and PBM work fine.

for format in (png ppm pgm pbm pam)
  convert img.dng img.$format
  flif img.$format img.$format.flif
end

Only PGM and PBM return, the rest receive SIGKILL from my OS.

Also I recall this working last night so I suspect a recent commit may have caused issues.

FLIF-hub commented 9 years ago

Can you paste the output it gives when you use lots of verbosity? (flif -vvvvvvvv)

bobobo1618 commented 9 years ago

Sure:

  _____  __  (__) _____
 (___  ||  | |  ||  ___)   FLIF 0.1 [2 October 2015]
  (__  ||  |_|__||  __)    Free Lossless Image Format
    (__||______) |__)      (c) 2010-2015 J.Sneyers & P.Wuille, GNU GPL v3+

Loading input file: IMG_0772.png  
Input: 4290x2856, channels: 3, depth: 16 bit
Transforms: YIQ, BND[0:232..65535][1:10117..128595][2:29438..82138]
FLIF-hub commented 9 years ago

Is that all the output? Must be a bug in ACB then. I thought I disabled ACB on 16-bit images, but it looks like it is trying to construct huge amounts of color buckets...

bobobo1618 commented 9 years ago

Yep, turned off ACB and it seems to be fine. Weird that it does it on PNGs and not PGMs and such though.

On Sun, Oct 4, 2015 at 11:22 AM Jon Sneyers notifications@github.com wrote:

Is that all the output? Must be a bug in ACB then. I thought I disabled ACB on 16-bit images, but it looks like it is trying to construct huge amounts of color buckets...

— Reply to this email directly or view it on GitHub https://github.com/jonsneyers/FLIF/issues/33#issuecomment-145370969.

FLIF-hub commented 9 years ago

That's because PGM and PBM don't have color, so the auto-color-buckets transform is disabled for those images anyway. I should generalize ACB to arbitrary bit depth (right now, it will create an amount of buckets that is reasonable for 8-bit images but insane for 16-bit images), but I'm a bit lazy now so I'll just disable it for high bit depth images. The next commit should do just that.