search

dropbox / lepton

Lepton is a tool and file format for losslessly compressing JPEGs by an average of 22%.
https://blogs.dropbox.com/tech/2016/07/lepton-image-compression-saving-22-losslessly-from-images-at-15mbs/
Apache License 2.0
5.01k stars 355 forks source link

Rare lepton crash on decompression #146

Closed leijurv closed 3 years ago

leijurv commented 3 years ago

Similar to my previous issue #129, Lepton randomly failed this time on decompression, but rerunning it on that specific file worked fine:

2021/01/03 08:44:26 Expected hash for this entry is ccbededbfe83c9a071af0fc5a1e212d9c2eedc283c0ac8fe404c0f2603288c72, decompressing...
lepton v1.0-1.2.1-183-g3d1bc19
prctl(SECCOMP): Unknown error 524
panic: entry was wrong size

Of those four log lines, the first and last are from my code, and the middle two are from lepton.

Here is the file in question: https://www.dropbox.com/s/ld4pn9l0dh1qgvp/leptonissue.lep?dl=1

It works fine normally: cat leptonissue.lep | lepton - | shasum -a 256 gives the hash as expected (ccbeded...). I will rerun this in a loop and see if I can make it crash again.

danielrh commented 3 years ago

I ran this 10,000 times without issue Unlike the other problem, this seems to be an issue with your linux kernel: https://discuss.linuxcontainers.org/t/lxd-cannot-scale-over-597-containers-seccomp-errors/7046 perhaps upgrading to linux 5.4 could solve your issue or reducing the number of simultaneous lepton invocations to match your kernel configuration You could also run -unjailed, but this is not recommended as lepton is written in C++, a language which is conducive to undefined behaviors.

However for #129 I would really love one example file I could be certain was failing, so that when I apply thread memory pressure, I'm sure it will at least trigger the code paths that could result in an incorrect success

leijurv commented 3 years ago

I'm suspicious of that because this code only ran one lepton subprocess at a time sequentially. Also I am running this on Ubuntu 20.04 which is already Linux 5.4. Thank you for taking the time to look though I appreciate it.

For #129 I have added more logging so that if it happens again I will be able to let you know specifically what it was :) Sorry for wasting your time on this what looks like a false alarm.