dropbox / lepton

Lepton is a tool and file format for losslessly compressing JPEGs by an average of 22%.
https://blogs.dropbox.com/tech/2016/07/lepton-image-compression-saving-22-losslessly-from-images-at-15mbs/
Apache License 2.0
5.01k stars 355 forks source link

Cannot pass "make check", then encoding fails as well #78

Closed xingxu0 closed 7 years ago

xingxu0 commented 7 years ago

Using single core compile on my Ubuntu machine (multi-core compile shows similar results), test-suites.log copied below. When encoding a JPEG, it says:

lepton v1.0-1.2.1-37-g7e9ef87 Illegal instruction (core dumped) SHORT_READ

=================================== lepton 0.01: ./test-suite.log

TOTAL: 39

PASS: 8

SKIP: 0

XFAIL: 0

FAIL: 31

XPASS: 0

ERROR: 0

.. contents:: :depth: 2

FAIL: test_suite/test_invariants

FAIL: test_suite/test_baseline

./lepton -hugepages -decode -recodememory=24M - (NULL) ./lepton -hugepages -recode -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.278933 encode process exit: 0.279001 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_misc

Loading 95820 ./lepton -hugepages -decode -recodememory=24M - (NULL) ./lepton -hugepages -recode -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.308736 encode process exit: 0.308841 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_iphone

Loading 2236391 ./lepton -hugepages -decode -recodememory=24M - (NULL) ./lepton -hugepages -recode -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.538831 encode process exit: 0.538909 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_phone_outdoor

Loading 2670607 ./lepton -hugepages -decode -recodememory=24M - (NULL) ./lepton -hugepages -recode -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.510263 encode process exit: 0.510344 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_truncate

Loading 4194304 ./lepton -hugepages -decode -recodememory=24M - (NULL) ./lepton -hugepages -recode -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.626987 encode process exit: 0.627035 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_android_lowmem

Loading 129432 ./lepton -hugepages -decode -singlethread -threadmemory=0M -recodememory=24M - (NULL) ./lepton -hugepages -recode -singlethread -memory=18M -threadmemory=0M -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.308864 encode process exit: 0.308906 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_SLR

Loading 3164341 ./lepton -hugepages -decode -memory=256M -recodememory=24M - (NULL) ./lepton -hugepages -recode -memory=256M -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.671253 encode process exit: 0.671327 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_progressive

Loading 101316 ./lepton -hugepages -decode -allowprogressive -recodememory=24M - (NULL) ./lepton -hugepages -recode -allowprogressive -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.312892 encode process exit: 0.312958 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_hq

Loading 23658057 ./lepton -hugepages -decode -memory=360M -threadmemory=24M -recodememory=87M - (NULL) ./lepton -hugepages -recode -memory=360M -threadmemory=24M -recodememory=87M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 2.087437 encode process exit: 2.087552 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/forktester.py

SHORT_READ filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG ('written ', <open file '/tmp/2419496692-a67c-9e28-b2c6ab447028-7e.iport', mode 'wb' at 0x2ac06e516db0>) closed (2236391, 0) ('written ', <open file '/tmp/9b9941ab37-5141-a304-78f4174bc601-b4.iport', mode 'wb' at 0x2ac06e516ed0>) ('xclosed ', <closed file '/tmp/9b9941ab37-5141-a304-78f4174bc601-b4.iport', mode 'wb' at 0x2ac06e516ed0>) Traceback (most recent call last): File "./test_suite/forktester.py", line 57, in assert (ojpg == jpg) AssertionError filetype of file "-" is unknown4096 bytes needed to decompress this file filetype of file filetype of fifiletype of file "-" is unknownfiletype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEGUNSUPPORTED_JPEGU4096 bytes need UNSUPPORTED_JPEG4096 bytes needed to decompress this file UNSUPPORTED_JPEG EARLY_EXIT

FAIL: test_suite/sockettester.py

SHORT_READ Child 17404 exited with signal 4 filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG Child 17417 exited with code 42 ('encode time ', 0.4703710079193115) (2236391, 0) 0 2236391 Traceback (most recent call last): File "./test_suite/sockettester.py", line 134, in test_compression('./lepton') File "./test_suite/sockettester.py", line 109, in test_compression assert (ojpg == jpg) AssertionError

FAIL: src/lepton/test_custom_table.sh

Using default file ./src/lepton/../../images/androidprogressive.jpg lepton v1.0-1.2.1-37-g7e9ef87 Illegal instruction (core dumped) SHORT_READ lepton v1.0-1.2.1-37-g7e9ef87 Illegal in lepton v1.0-1.2.1-37-g7e9ef87 filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG d41d8cd98f00b204e9800998ecf8427e /tmp/temp.qqEG71 7fa3cb503d17534e7658ee8da9145f8d ./src/lepton/../../images/androidprogressive.jpg Files /tmp/temp.qqEG71 and ./src/lepton/../../images/androidprogressive.jpg differ compression_model /tmp/temp.eM2yDB test_model /tmp/temp.d6byWD compressed_lepton /tmp/temp.jDzMfV roundtrip /tmp/temp.qqEG71

FAIL: test_suite/test_baseline_unjailed

Inject failure level is 1 so is it encode args? 1 or decode_args? 0 ./lepton -hugepages -decode -injectsyscall=1 -singlethread -unjailed -recodememory=24M - (NULL) ./lepton -hugepages -recode -singlethread -unjailed -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 START ACHIEVED 1484780945 425095 SHORT_READ Timing encode: 0.279521 encode process exit: 0.279598 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknownTS_MAIN (0) 0.000000 TS_DONE (0) 0.664786 4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_baseline_unjailed_thread

Inject failure level is 2 so is it encode args? 1 or decode_args? 0 ./lepton -hugepages -decode -injectsyscall=2 -singlethread -unjailed -recodememory=24M - (NULL) ./lepton -hugepages -recode -singlethread -unjailed -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 START ACHIEVED 1484780946 120528 SHORT_READ Timing encode: 0.298649 encode process exit: 0.298724 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknownTS_MAIN (0) 0.000000 TS_DONE (0) 0.682726 4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_baseline_unjailed_decode

Inject failure level is 3 so is it encode args? 0 or decode_args? 1 ./lepton -hugepages -decode -singlethread -unjailed -recodememory=24M - (NULL) ./lepton -hugepages -recode -injectsyscall=1 -singlethread -unjailed -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 START ACHIEVED 1484780946 835131 SHORT_READ Timing encode: 0.286378 encode process exit: 0.286408 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknownTS_MAIN (0) 0.000000 TS_DONE (0) 0.664060 4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_baseline_unjailed_decode_thread

Inject failure level is 4 so is it encode args? 0 or decode_args? 1 ./lepton -hugepages -decode -singlethread -unjailed -recodememory=24M - (NULL) ./lepton -hugepages -recode -injectsyscall=2 -singlethread -unjailed -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 START ACHIEVED 1484780947 703075 SHORT_READ Timing encode: 0.288978 encode process exit: 0.289053 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknownTS_MAIN (0) 0.000000 TS_THREAD_STARTED (0) 0.001297 TS_DONE (0) 0.680743 4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_seccomp_decode_main

Inject failure level is 3 so is it encode args? 0 or decode_args? 1 ./lepton -hugepages -decode -singlethread -recodememory=24M - (NULL) ./lepton -hugepages -recode -injectsyscall=1 -singlethread -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.281846 encode process exit: 0.281902 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown

FAIL: test_suite/test_seccomp_decode_thread

Inject failure level is 4 so is it encode args? 0 or decode_args? 1 ./lepton -hugepages -decode -singlethread -recodememory=24M - (NULL) ./lepton -hugepages -recode -injectsyscall=2 -singlethread -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.269800 encode process exit: 0.269875 test_suite/timing_driver.cc:399: ret > 0

FAIL: test_suite/test_truncate_lowmem

Loading 4194304 ./lepton -hugepages -decode -singlethread -threadmemory=0M -recodememory=24M - (NULL) ./lepton -hugepages -recode -singlethread -memory=18M -threadmemory=0M -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.581869 encode process exit: 0.581946 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_nofsync

Loading 5649 ./lepton -hugepages -recode -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.286963 encode process exit: 0.287045 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_colorswap

Loading 308 ./lepton -hugepages -decode -recodememory=24M - (NULL) ./lepton -hugepages -recode -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.283509 encode process exit: 0.283556 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_odd_rst

Loading 35243 ./lepton -hugepages -decode -recodememory=24M - (NULL) ./lepton -hugepages -recode -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.277855 encode process exit: 0.277901 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_trailing_header

Loading 96104 ./lepton -hugepages -decode -recodememory=24M - (NULL) ./lepton -hugepages -recode -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.292008 encode process exit: 0.292064 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_trailing_rst

Loading 57524 ./lepton -hugepages -decode -recodememory=24M - (NULL) ./lepton -hugepages -recode -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.291617 encode process exit: 0.291650 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_legacy.sh

lepton v1.0-1.2.1-37-g7e9ef87 Illegal instruction (core dumped)

FAIL: test_suite/test_gray2sf

Loading 100309 ./lepton -hugepages -decode -recodememory=24M - (NULL) ./lepton -hugepages -recode -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.301796 encode process exit: 0.301879 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_2nd_block.sh

lepton v1.0-1.2.1-37-g7e9ef87 lepton v1.0-1.2.1-37-g7e9ef87 Illegal in filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_3rd_block.sh

lepton v1.0-1.2.1-37-g7e9ef87 lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_last_block.sh

lepton v1.0-1.2.1-37-g7e9ef87 lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

FAIL: test_suite/test_truncated_zero_run

Loading 100971 ./lepton -hugepages -decode -recodememory=24M - (NULL) ./lepton -hugepages -recode -recodememory=24M - (NULL) Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 Huge pages unsupported: falling back to ordinary pages lepton v1.0-1.2.1-37-g7e9ef87 SHORT_READ Timing encode: 0.286810 encode process exit: 0.286845 test_suite/timing_driver.cc:399: ret > 0filetype of file "-" is unknown4096 bytes needed to decompress this file UNSUPPORTED_JPEG

danielrh commented 7 years ago

the illegal instruction seems to indicate that maybe it was compiled for an unsupported architecture. Does your system have sse4.2? does it have avx or avx2? the compile flags may require those settings unless you build with --scalar on the emscripten branch

danielrh commented 7 years ago

So maybe try this particular git hash and see if it works for you--if so we can try to merge it into the master branch

https://github.com/dropbox/lepton/commit/d2f87e59f4883d774948a487871d0f5aff7aa5ca

or the main emscripten branch https://github.com/dropbox/lepton/tree/emscripten

Do either of those help you build?

danielrh commented 7 years ago

added a new configure option ./configure --disable-vectorization that should fix things for you on the master branch