Yoshimi / yoshimi

A sophisticated soft-synth originally forked from ZynAddSubFX V2.4.0 in 2009 by Alan Calvert, and still in continuous development - This is also mirrored at http://sourceforge.net/projects/yoshimi/ : Current news is at http://sourceforge.net/p/yoshimi/news/ : Our email discussion list is: http://www.freelists.org/list/yoshimi and here is our website
http://yoshimi.github.io/
Other
238 stars 39 forks source link

Yoshimi crashes when there's too many (open) files in Ardour #201

Open lminiero opened 6 months ago

lminiero commented 6 months ago

This is an old issue I bumped into some time ago, and it seems more a problem in libfltk than in Yoshimi itself, but I thought I'd open an issue anyway, just in case you were not aware of it.

When a session that has too many open files crashed, you won't be able to open the session again either, since it will crash in the same point: the only fix is to manually edit the Ardour session file, remove the Yoshimi LV2 instances, start again, and then clean unused sources to reduce the number of open files before re-adding Yoshimi again. I don't know how frequent having too many open files is, but my Ardour projects have a ton of tracks, and being a sloppy player I do many takes of my parts. Audio captures that were removed from Ardour still live there, to allow for undo's (more context here, where I originally reported the problem a few years ago).

This is what Ardour gives me when crashing at startup (I snipped all the Ardour calls before, since they're not relevant):

*** bit out of range 0 - FD_SETSIZE on fd_set ***: terminated

Thread 1 "ArdourGUI" received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, 
    no_tid=no_tid@entry=0) at pthread_kill.c:44
44        return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt
c#0  __pthread_kill_implementation
    (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0)
    at pthread_kill.c:44
#1  0x00007ffff47cf8f3 in __pthread_kill_internal
    (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007ffff477eafe in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/posix/raise.c:26
#3  0x00007ffff476787f in __GI_abort () at abort.c:79
#4  0x00007ffff476860f in __libc_message
    (fmt=fmt@entry=0x7ffff48e22e6 "*** %s ***: terminated\n")
    at ../sysdeps/posix/libc_fatal.c:150
#5  0x00007ffff4863a79 in __GI___fortify_fail
    (msg=msg@entry=0x7ffff48e7398 "bit out of range 0 - FD_SETSIZE on fd_set")
    at fortify_fail.c:24
#6  0x00007ffff48639c5 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:26
#7  0x00007ffe8eb77d60 in Fl::remove_fd (n=n@entry=1149, events=events@entry=1)
    at /usr/src/debug/fltk-1.3.8-6.fc38.x86_64/src/Fl_x.cxx:187
#8  0x00007ffe8eb77e15 in Fl::add_fd
    (n=1149, events=events@entry=1, cb=cb@entry=0x7ffe8eb77b80 <thread_awake_cb(int, void*)>, v=v@entry=0x0)
    at /usr/src/debug/fltk-1.3.8-6.fc38.x86_64/src/Fl_x.cxx:115
#9  0x00007ffe8eb77fb6 in Fl::lock ()
    at /usr/src/debug/fltk-1.3.8-6.fc38.x86_64/src/Fl_lock.cxx:351
--Type <RET> for more, q to quit, c to continue without paging--c
#10 0x00007ffe4768e329 in YoshimiLV2Plugin::instantiate
    (desc=0x7ffe47aec460 <yoshimi_lv2_desc>, sample_rate=48000, bundle_path=0x555582f3a7d0 "/usr/lib64/lv2/yoshimi.lv2/", features=0x555582f3a750)
    at /usr/src/debug/yoshimi-2.2.3-1.fc38.x86_64/src/LV2_Plugin/YoshimiLV2Plugin.cpp:427
#11 0x00007ffff59bcc56 in lilv_plugin_instantiate () at /lib64/liblilv-0.so.0
...

The problem is that libfltk is using FD_SET internally, but that's known to be severely limited, since due to the mask it uses, it's limited to 1024 descriptors. When too many files are opened (more than that), the call will have undefined behaviour and break. This is why all network applications (or applications using FD_SET for file descriptors) use poll instead.

Again, this is an error in libfltk, not Yoshimi directly, but since Yoshimi is using that library, it's impacted by this problem, which in turn brings everything down.

abrolag commented 6 months ago

On Sun, 10 Mar 2024 01:52:29 -0800 Lorenzo Miniero @.***> wrote:

This is an old issue I bumped into some time ago, and it seems more a problem in libfltk than in Yoshimi itself, but I thought I'd open an issue anyway, just in case you were not aware of it.

When a session that has too many open files crashed, you won't be able to open the session again either, since it will crash in the same point: the only fix is to manually edit the Ardour session file, remove the Yoshimi LV2 instances, start again, and then clean unused sources to reduce the number of open files before re-adding Yoshimi again. I don't know how frequent having too many open files is, but my Ardour projects have a ton of tracks, and being a sloppy player I do many takes of my parts. Audio captures that were removed from Ardour still live there, to allow for undo's (more context here, where I originally reported the problem a few years ago).

This is what Ardour gives me when crashing at startup (I snipped all the Ardour calls before, since they're not relevant):

*** bit out of range 0 - FD_SETSIZE on fd_set ***: terminated

Thread 1 "ArdourGUI" received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, ***@***.***=6, 
   ***@***.***=0) at pthread_kill.c:44
44       return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) bt
c#0  __pthread_kill_implementation
   (threadid=<optimized out>, ***@***.***=6, ***@***.***=0)
   at pthread_kill.c:44
#1  0x00007ffff47cf8f3 in __pthread_kill_internal
   (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007ffff477eafe in __GI_raise ***@***.***=6)
   at ../sysdeps/posix/raise.c:26
#3  0x00007ffff476787f in __GI_abort () at abort.c:79
#4  0x00007ffff476860f in __libc_message
   ***@***.***=0x7ffff48e22e6 "*** %s ***: terminated\n")
   at ../sysdeps/posix/libc_fatal.c:150
#5  0x00007ffff4863a79 in __GI___fortify_fail
   ***@***.***=0x7ffff48e7398 "bit out of range 0 - FD_SETSIZE on fd_set")
   at fortify_fail.c:24
#6  0x00007ffff48639c5 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:26
#7  0x00007ffe8eb77d60 in Fl::remove_fd ***@***.***=1149, ***@***.***=1)
   at /usr/src/debug/fltk-1.3.8-6.fc38.x86_64/src/Fl_x.cxx:187
#8  0x00007ffe8eb77e15 in Fl::add_fd
   (n=1149, ***@***.***=1, ***@***.***=0x7ffe8eb77b80 <thread_awake_cb(int, void*)>, ***@***.***=0x0)
   at /usr/src/debug/fltk-1.3.8-6.fc38.x86_64/src/Fl_x.cxx:115
#9  0x00007ffe8eb77fb6 in Fl::lock ()
   at /usr/src/debug/fltk-1.3.8-6.fc38.x86_64/src/Fl_lock.cxx:351
--Type <RET> for more, q to quit, c to continue without paging--c
#10 0x00007ffe4768e329 in YoshimiLV2Plugin::instantiate
   (desc=0x7ffe47aec460 <yoshimi_lv2_desc>, sample_rate=48000, bundle_path=0x555582f3a7d0 "/usr/lib64/lv2/yoshimi.lv2/", features=0x555582f3a750)
   at /usr/src/debug/yoshimi-2.2.3-1.fc38.x86_64/src/LV2_Plugin/YoshimiLV2Plugin.cpp:427
#11 0x00007ffff59bcc56 in lilv_plugin_instantiate () at /lib64/liblilv-0.so.0
...

The problem is that libfltk is using FD_SET internally, but that's known to be severely limited, since due to the mask it uses, it's limited to 1024 descriptors. When too many files are opened (more than that), the call will have undefined behaviour and break. This is why all network applications (or applications using FD_SET for file descriptors) use poll instead.

Again, this is an error in libfltk, not Yoshimi directly, but since Yoshimi is using that library, it's impacted by this problem, which in turn brings everything down.

I've not come across anything like this before, but then I don't use Ardour apart from basic testing. I doubt there is much -if anything- we can do about it. However, what version of FLTK are you using? I'm on V1.3.8 and I know V1.3.9 is available, so it may have been addressed there. Also the FLTK development branch is V1.4.0 and may have resolved this - but you would have to build and install that yourself (removing V1.3.x).

Meanwhile I'll ask around the other devs.

-- Will J Godfrey {apparently now an 'elderly'} https://willgodfrey.bandcamp.com/ http://yoshimi.github.io Say you have a poem and I have a tune. Exchange them and we can both have a poem, a tune, and a song.

lminiero commented 6 months ago

Thanks for the quick answer! I'm on 1.3.8 too, but looking at their github repo, the latest branch still uses FD_SET, and none of the existing pull requests seem to address a change on that.

I think the fact I'm using Ardour is only partly relevent: actually, it's only "accelerating" the issue due to my large projects with a ton of tracks, which causes a lot of files to be opened, thus more easily causing file descriptors to go beyond 1024 in Yoshimi/fltk. In smaller projects (and with better players that don't have to record their part dozens of time :smile: ) it's unlikely to happen.

abrolag commented 5 months ago

On Sun, 10 Mar 2024 05:43:34 -0700 Lorenzo Miniero @.***> wrote:

Thanks for the quick answer! I'm on 1.3.8 too, but looking at their github repo, the latest branch still uses FD_SET, and none of the existing pull requests seem to address a change on that.

I think the fact I'm using Ardour is only partly relevent: actually, it's only "accelerating" the issue due to my large projects with a ton of tracks, which causes a lot of files to be opened, thus more easily causing file descriptors to go beyond 1024 in Yoshimi/fltk. In smaller projects (and with better players that don't have to record their part dozens of time :smile: ) it's unlikely to happen.

With further thought this may be a Yoshimi issue :( If Ardour is holding on to the (currently) unused instance IDs, Yoshimi will create new IDs. However, we never thought anyone would want more than 31 instances, simply because no processor for the foreseeable future could actually run that many! We use the bits of an integer to mark which are in use, so we could change to a 64 bit storage, but that probably only delays the issue. I'll try artificially reducing the limit and see exactly what Yoshimi does when trying to create another - it should simply refuse to start the new one.

-- Will J Godfrey {apparently now an 'elderly'} https://willgodfrey.bandcamp.com/ http://yoshimi.github.io Say you have a poem and I have a tune. Exchange them and we can both have a poem, a tune, and a song.