Xvideo driver does leak the shared memory segments

The Xvideo driver creates some segments of shared memory with shmat() but
never removes these segments with shmctl(,IPC_RMID,). Though the processes
unmap these segments, the kernel continues to retain these segments. This
can eventually cause the emulator to crash.

What steps will reproduce the problem?
1. Set the hardware acceleration to XVideo.
2. Repeatedly start and quit snes9x-gtk.
3. Use the ipcs command to list the leaked segments.
4. Eventually, snes9x-gtk always dumps core.
5. Editing ~/.snes9x/snes9x.xml, change "hw_accel" from "2" (XVideo) to "0"
(no acceleration). Then snes9x-gtk does not crash.
6. Set the hardware acceleration to XVideo again, then snes9x-gtk dumps
core when applying the preferences.
7. Use the ipcrm -m command to manually remove the leaked segments. The
XVideo driver works again. Alternatively, reboot the operating system to
remove the leaked segments.

I know not the exact cause of the core dump. My guess that the leaked
segments reach some limit to cause shmat() to fail. Then snes9x-gtk never
checks if shmat() failed, so it tries to use the new segment and dumps core.

I first produced the crashes with snes9x-gtk patch 65. I upgraded to patch
67 before I report this issue. I can confirm that snes9x-gtk patch 67 leaks
one segment if I start snes9x-gtk (without loading any ROM) and close the
window. When ipcs lists 18 segments, then snes9x-gtk crashes on my system.

If I then play a video, then VLC media player reports "xvideo video output
error: cannot allocate shared image data (Cannot allocate memory)" but
plays the video through some other way. Somehow, ipcs lists 22 segments
while VLC runs, and 19 segments after VLC exits.

My operating system is OpenBSD.

$ uname -a
OpenBSD ghostborough.local 4.5 GENERIC#2013 macppc
$ ipcs
...
Shared Memory:
T       ID     KEY        MODE       OWNER    GROUP
...
m   131085          0 --rwarwarwa  kernigh  kernigh
m   131087          0 --rwarwarwa  kernigh  kernigh
m   131089          0 --rwarwarwa  kernigh  kernigh
...
$ ipcrm -m 131085 -m 131087 -m 131089 ...
$ ipcs | grep ^m | sed 's,^m *,,' | cut -d' ' -f1 | xargs -n1 ipcrm -m
ipcrm: shmid(262151): Invalid argument
ipcrm: shmid(655370): Invalid argument
ipcrm: shmid(1376267): Invalid argument

That last command deletes all but 3 segments and allows snes9x-gtk to not
crash.

I am not familiar with Xvideo and Shm, but the source code of VLC suggests
how to call shmctl. In vlc-0.8.6i/modules/video_output/x11/xcommon.c:

    /* The order of operations is correct */
#ifdef HAVE_SYS_SHM_H
    if( p_vout->p_sys->i_shm_opcode )
    {
        XShmDetach( p_vout->p_sys->p_display, &p_pic->p_sys->shminfo );
        IMAGE_FREE( p_pic->p_sys->p_image );

        shmctl( p_pic->p_sys->shminfo.shmid, IPC_RMID, 0 );
        if( shmdt( p_pic->p_sys->shminfo.shmaddr ) )
        {
            msg_Err(...

I have attached a diff -u gtk_display_driver_xv.cpp{.orig,} that inserts
similar shmctl calls into snes9x-gtk. This diff prevents that snes9x-gtk
leak memory segments. (This diff does not prevent a core dump if shmat fails.)
Original issue reported on code.google.com by kern...@gmail.com on 1 Mar 2009 at 2:11
Attachments:
gtk_display_driver_xv.cpp.diff
mchoccac / snes9x-gtk

Xvideo driver does leak the shared memory segments #2
