dropbox / lepton

Lepton is a tool and file format for losslessly compressing JPEGs by an average of 22%.
https://blogs.dropbox.com/tech/2016/07/lepton-image-compression-saving-22-losslessly-from-images-at-15mbs/
Apache License 2.0
5.01k stars 355 forks source link

Lepton on Linux < 3.1 & OOM with big image #61

Closed fxrlv closed 7 years ago

fxrlv commented 8 years ago

Hello, I am trying to build this application on

Linux centos6 2.6.32-358.el6.x86_64

But I have next one error

src/lepton/jpgcoder.cc:52:27: fatal error: linux/seccomp.h: No such file or directory

Okey, I've installed a 'kernel-devel' package on the machine and compile with next one flags

-I /usr/src/kernels/2.6.32-642.3.1.el6.x86_64/include

And I've got next one errors

src/io/seccomp-bpf.hh:69:29: error: ‘SECCOMP_RET_KILL’ was not declared in this scope src/io/seccomp-bpf.hh:76:29: error: ‘SECCOMP_RET_ALLOW’ was not declared in this scope src/io/Seccomp.cc:83:31: error: ‘SECCOMP_MODE_STRICT’ was not declared in this scope src/io/Seccomp.cc:97:35: error: ‘SECCOMP_MODE_FILTER’ was not declared in this scope

I've found that kernel's headers have no this define

fgrep "SECCOMP_RET_KILL" /usr/src/kernels/2.6.32-642.3.1.el6.x86_64 -R

As I've found, maybe it's wrong, this define was appeared in v3.1+ kernel

It would be nice if I can fully disable the SECCOMP on a build phase

Okey, next one question:

In your article

https://blogs.dropbox.com/tech/2016/07/lepton-image-compression-saving-22-losslessly-from-images-at-15mbs/

You said that all operations with image are under 24 megabytes. So, what is it? screen shot 2016-08-12 at 11 50 16

./lepton -singlethread -skipvalidation /tmp/world.topo.bathy.200407.3x21600x10800.jpg -memory=4096M image size: 27201049 bytes

If I run without '-memory=4096M' I have next one assert

Assertion failed: (false && "Out of memory error"), function custom_calloc

In 'Release' Lepton version I have same problem with memory usage

Image is from NASA site

http://eoimages.gsfc.nasa.gov/images/imagerecords/73000/73751/world.topo.bathy.200407.3x21600x10800.jpg

danielrh commented 8 years ago

I will try to add a SECCOMP disable flag that will also remove any memory limitations. With SECCOMP, you need to pre-allocate memory before parsing hostile data--so we wouldn't be able to guess the appropriate memory limit. With -unjailed mode (or command line disabled seccomp) we should be able to lean on the system allocator and get memory as needed--next version will do this

danielrh commented 7 years ago

This is now fixed on master branch. I'll cut a new release in a week or two

ssbanerje commented 7 years ago

Is this available on the CMake build? How do I disable seccomp?