Proper Xenomai inter-process communication (IPC)

[giuliomoro]

The issue is that staticMAudioSync->Signal(); (which in turns calls std::condition_variable_any::notify_one ) was causing a mode switch in the audio thread. We removed the mode switch from the audio thread by moving it to another thread:

    // this avoids Xenomai mode switches in the audio thread ...
    Bela_scheduleAuxiliaryTask(mAudioSyncSignalTask);
}

void SC_BelaDriver::staticMAudioSyncSignal(){
    // ... but mode switches are still happening here, in a lower priority thread.
    // FIXME: this triggers a mode switch in Xenomai.
    staticMAudioSync->Signal();
    rt_task_suspend(rt_task_self());
}

, still this is inelegant, non-deterministic and adds to the CPU load.

The proper way around this is to rewrite common/SC_SyncCondition.h and common/SC_lock.h so that they use Xenomai-specific IPC functions instead of condition_variable_any and std::mutex. The discussion here shows a possible method using RT_COND, rt_cond_wait, rt_cond_signal, RT_MUTEX, rt_mutex_acquire, rt_mutex_release

[sensestage]

ok - this is pretty lowlevel stuff. In principle it should of course be possible to do it; we'll have to look into that together then, I guess.

[sensestage]

Does it also cause problems with boost::sync::semaphore(0)? It seems that that is where the quit message fails. (see issue #9).

[giuliomoro]

not sure. I don't see why it should.

[giuliomoro]

It may be related to having heavy CPU load in the audio thread?

[sensestage]

No, the quit message sometimes even fails when there are no synths running on the server.

[sensestage]

going into detail in issue #9

[giuliomoro]

I don't think this inherently causes any problems with boost::sync::semaphore(0).

The only problem I could see is that the last call to

staticMAudioSync->Signal();

(which originally was mAudioSync.Signal())

does not get executed because the auxiliary thread it runs in is terminated before it gets executed after the last call to SC_BelaDriver::BelaAudioCallback.

Thinking of how to reduce CPU usage for this operation, the suggestion above is not going to work because using xenomai's condition variables forces the thread to switch to primary mode, and then it would switch back to secondary mode once resumed, so: using Xenomai's condition variables would not remove mode switches, would only move them to another thread, with no advantage for us. Similar results would be obtained using Xenomai's semaphores or queues.

The only path provided by Xenomai for communication between real-time and non-real-time threads seems to be "pipes", which uses a pseudo-device file descriptor at the non-real-time end. http://www.xenomai.org/documentation/trunk/html/api/group__pipe.html But I have the impression that this would cause more overhead than the current implementation.

Here, for instance, we used Xenomai's pipes https://github.com/BelaPlatform/Bela/blob/2b9b0bf04fd07b8ada325fd27da7a6c1685a60b4/core/Midi.cpp

[giuliomoro]

Back to the point, what mAudioSync->Signal(); does is waking up a low-priority thread, which does (in server/scsynth/SC_CoreAudio.cpp):

// send /tr messages
 trigfifo->Perform();
// send node reply messages
nodereplyfifo->Perform();
// send node status messages
nodeendfifo->Perform();
// free GraphDefs
deletegraphfifo->Perform();
// perform message
 mFromEngine.Perform();

Now, when the CPU load is low, there are occasional "dropouts", as in: the number of times this thread is actually woken up is smaller than the number of times it is signalled. This makes sense, if you think that this is a low-prio thread and should not necessarily have to wake up every time the audio callback is terminated (as we are requesting it to do by Signal()ling it at the end of each audio callback!). This is probably ok for large block sizes. It is not clear whether the thread does not wake up because it is already awake, still busy with its tasks, or whether it is not woken up at all because Linux decides to go and do something else instead. It seems that when CPU usage increases to about 70% for the audio thread, at least 30% of the wake ups are lost

Either way - without even looking at the internal code of all the above tasks - I think it is safe to assume that - by design - they a) are all interruptible and thread-safe, and b) they can run a reasonably small number of times per seconds, even smaller than Fs/blocksize per second (which - again - is the number of times the thread is resumed). (I say I assume that because otherwise all the SC programs would always be one step away from collapsing).

This said, in principle we should be able to simply set this task to sleep for, e.g.: 1ms so that it wakes itself up without need for synchronization with the audio callback. BUT when I proposed a similar solution in https://github.com/sensestage/supercollider/commit/50e26c54fa8992445b132afafbdef3607ed5ed65, we were often getting "synthdef not found" errors, as if the synthdefs were not loaded by the server, which is why we ended up in the current situation. I guess it makes sense give it another try.