Fl_JPEG_Image locks up on macOS when given garbage data

ioan-chera / eureka-editor

Eureka is a cross-platform map editor for the DOOM engine games.

http://eureka-editor.sourceforge.net/

64 stars 12 forks source link

Fl_JPEG_Image locks up on macOS when given garbage data #141

Closed ioan-chera closed 1 year ago

ioan-chera commented 1 year ago

Using FLTK 1.3.8 and testing on Mac (unknown on Windows), giving invalid JPEG data will result in an infinite loop of printing this error:

JPEG data is too large or contains errors!

On Ubuntu, as seen in Actions, it doesn't lock up.

ioan-chera commented 1 year ago

Turns out it's a case of compiler over-optimization around setjmp. Adding volatile pointers and using them inside setjmp fixes the issue, but it's not my code. It happens in the constructors of Fl_JPEG_Image.

ioan-chera commented 1 year ago

Solved by applying a patch on FLTK before building it. If you're on Mac, make sure you delete the FLTK archive and dir from under build/src before you rebuild!