32 bit gifsicle-1.90 segfaults when resizing a big GIF file

[stefantalpalaru]

dmesg output

[20207289.259665] traps: gifsicle[31824] general protection ip:806b22a sp:ffe76f40 error:0
[20207289.259669]  in gifsicle[8048000+3a000]

gdb backtrace

(gdb) r
Starting program: /usr/bin/gifsicle --resize-method lanczos3 --resize 500x500 example.gif
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
ksscreen_apply (ks=0x809a1d0, gfi=<optimized out>, kss=0x718) at xform.c:518
518 xform.c: No such file or directory.
(gdb) bt
#0  ksscreen_apply (ks=0x809a1d0, gfi=<optimized out>, kss=0x718) at xform.c:518
#1  scale_image_prepare (sctx=sctx@entry=0xffffce80) at xform.c:672
#2  0x0806b64a in scale_image_data_weighted (sctx=0xffffce80, gfo=0xffffcdb0, weightf=<optimized out>, radius=3) at xform.c:1068
#3  0x0806cd7a in scale_image_data_lanczos3 (gfo=0xffffcdb0, sctx=0xffffce80) at xform.c:1125
#4  scale_image (sctx=sctx@entry=0xffffce80, method=method@entry=5) at xform.c:1198
#5  0x0806e646 in resize_stream (gfs=0x808c7b8, new_width=500, new_height=500, flags=0, method=5, scale_colors=0) at xform.c:1372
#6  0x0806ed36 in merge_and_write_frames (outfile=outfile@entry=0x0, f1=f1@entry=0, f2=f2@entry=-1) at gifsicle.c:1026
#7  0x08070113 in output_frames () at gifsicle.c:1105
#8  0x0804ad42 in main (argc=<optimized out>, argv=<optimized out>) at gifsicle.c:2173

strace

mmap2(NULL, 14401536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xf27d9000
brk(0x904e000)                          = 0x904e000
brk(0x907e000)                          = 0x907e000
sysinfo({uptime=20207110, loads=[267840, 261088, 257600], totalram=4107165, freeram=328001, sharedram=240699, bufferram=221904, totalswap=65535, freeswap=65471, procs=601, totalhigh=0, freehigh=0, mem_unit=4096}) = 0
mmap2(NULL, 51843072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xef668000
mmap2(NULL, 2002944, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xef47f000
mmap2(NULL, 2002944, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xef296000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=NULL} ---

notes

The problem does not appear on a 64 bit userspace. I know what you're thinking, but it's not feasible upgrading the 32 bit userspace where I'm seeing this.

file triggering the problem

https://user-images.githubusercontent.com/495550/33156969-094742f0-cfff-11e7-8ad1-4baa1af941b2.gif

[kohler]

Sorry, I can't replicate this with HEAD. Can you?

[stefantalpalaru]

Yes, I can replicate it with HEAD and get the same GDB backtrace. The software is configured with --disable-gifview and compiled with CFLAGS="-march=native -O3 -ggdb" CXXFLAGS="$CFLAGS" and gcc-6.4.0.

[Whissi]

I can reproduce on Gentoo x86 using test suite, see https://bugs.gentoo.org/651924 which also contains complete build.log.

Core was generated by `gifsicle -O2 --crop 25,0+36x36 --resize 100x100 x.gif'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  ksscreen_apply (ks=<optimized out>, gfi=<optimized out>, kss=0x47) at xform.c:518
518                     lineout[x] = sc_makekc(&ks[linein[x]]);
(gdb) bt
#0  ksscreen_apply (ks=<optimized out>, gfi=<optimized out>, kss=0x47) at xform.c:518
#1  scale_image_prepare (sctx=sctx@entry=0xbfb24f80) at xform.c:672
#2  0x00439a26 in scale_image_data_mix (gfo=0xbfb24ec0, sctx=0xbfb24f80) at xform.c:916
#3  scale_image (sctx=sctx@entry=0xbfb24f80, method=method@entry=2) at xform.c:1190
#4  0x0043b633 in resize_stream (gfs=0xf23860, new_width=100, new_height=100, flags=0, method=2, scale_colors=0)
    at xform.c:1372
#5  0x0043bc5f in merge_and_write_frames (outfile=0x0, f1=<optimized out>, f2=<optimized out>) at gifsicle.c:1026
#6  0x0043cefb in output_frames () at gifsicle.c:1105
#7  0x0041f735 in main (argc=<optimized out>, argv=<optimized out>) at gifsicle.c:2173

Core was generated by `gifsicle --resize 30x30 x.gif'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  ksscreen_apply (ks=<optimized out>, gfi=<optimized out>, kss=0x46) at xform.c:518
518                     lineout[x] = sc_makekc(&ks[linein[x]]);
(gdb) bt
#0  ksscreen_apply (ks=<optimized out>, gfi=<optimized out>, kss=0x46) at xform.c:518
#1  scale_image_prepare (sctx=sctx@entry=0xbfc19440) at xform.c:672
#2  0x004e1a26 in scale_image_data_mix (gfo=0xbfc19380, sctx=0xbfc19440) at xform.c:916
#3  scale_image (sctx=sctx@entry=0xbfc19440, method=method@entry=2) at xform.c:1190
#4  0x004e3633 in resize_stream (gfs=0x18fc418, new_width=30, new_height=30, flags=0, method=2, scale_colors=0)
    at xform.c:1372
#5  0x004e3c5f in merge_and_write_frames (outfile=0x0, f1=<optimized out>, f2=<optimized out>) at gifsicle.c:1026
#6  0x004e4efb in output_frames () at gifsicle.c:1105
#7  0x004c7735 in main (argc=<optimized out>, argv=<optimized out>) at gifsicle.c:2173

Core was generated by `gifsicle --resize=250x_ strip.gif'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  ksscreen_apply (ks=<optimized out>, gfi=<optimized out>, kss=0x51) at xform.c:518
518                     lineout[x] = sc_makekc(&ks[linein[x]]);
(gdb) bt
#0  ksscreen_apply (ks=<optimized out>, gfi=<optimized out>, kss=0x51) at xform.c:518
#1  scale_image_prepare (sctx=sctx@entry=0xbfed40f0) at xform.c:672
#2  0x00455a26 in scale_image_data_mix (gfo=0xbfed4030, sctx=0xbfed40f0) at xform.c:916
#3  scale_image (sctx=sctx@entry=0xbfed40f0, method=method@entry=2) at xform.c:1190
#4  0x00457633 in resize_stream (gfs=0x16c4418, new_width=250, new_height=0, flags=0, method=2, scale_colors=0)
    at xform.c:1372
#5  0x00457c5f in merge_and_write_frames (outfile=0x0, f1=<optimized out>, f2=<optimized out>) at gifsicle.c:1026
#6  0x00458efb in output_frames () at gifsicle.c:1105
#7  0x0043b735 in main (argc=<optimized out>, argv=<optimized out>) at gifsicle.c:2173

Core was generated by `gifsicle -O2 --careful --resize-method=mix --resize=200x20 in.gif'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0048bd6e in scale_image_data_mix (gfo=0xbfea7d10, sctx=0x0) at xform.c:946
946                     SCVEC_ADDVxF(sc[mix[i].xo], indata[mix[i].xi],
(gdb) bt
#0  0x0048bd6e in scale_image_data_mix (gfo=0xbfea7d10, sctx=0x0) at xform.c:946
#1  scale_image (sctx=sctx@entry=0xbfea7dd0, method=method@entry=2) at xform.c:1190
#2  0x0048d633 in resize_stream (gfs=0x187f828, new_width=200, new_height=20, flags=0, method=2, scale_colors=0)
    at xform.c:1372
#3  0x0048dc5f in merge_and_write_frames (outfile=0x0, f1=<optimized out>, f2=<optimized out>) at gifsicle.c:1026
#4  0x0048eefb in output_frames () at gifsicle.c:1105
#5  0x00471735 in main (argc=<optimized out>, argv=<optimized out>) at gifsicle.c:2173

[Whissi]

And this is still happening with 1.91.

[kohler]

I still don't see this happening. It does not happen on Travis in a 32-bit compile. I'm going to close this, but please reopen if it is still happening.