Latency compensation implementation when using direct monitoring

[jonjamcam]

I have a ME-25 Pedal connected to th PC via usb. The problem is that I can't disable the monitor input in the pedal and if I mute Jamtaba's channel it won't send it to the room. So I'm stuck with this duplicate sound that is really an unwanted FX.

So in ninjam and reaninjam I can solve this simply by muting the channel, but Jamaba is designed different, so I'm thinking for the average guy who just connect via usb there should be a preference to be able to mute the channels, but the sound to continue going to the room. Something like advanced (mute affects local channels only)

[Ezeer]

Are you using the ME-25 Pedal asio driver ?

[jonjamcam]

yes. It has a slight delay only noticeable when both overlap and no mix control

[Ezeer]

Can't u use an other driver for Jamtaba ? And use your phone output to the inputs of your alternate audio device ?

[jonjamcam]

Sure I can use my laptop asio4all, But I think many guitar players connect like this and listen to the jam via the pedal. Sounds really good and few connections and setup are needed.

windows 10 downloads the driver automatically and I use only 1usb cable to connect to the PC. That's pretty simple. No routing at all needed.

[Ezeer]

If you use tricks to make it work on other softawres that mean that your pedal was not designed to work like this . I understand that it would be nice to work like this ( plug and play ) , but your pedal was not designed to work that way .

You can ask the dev to find tricks , but it could be nice too to ask the pedal's company owner to fix that " detail " . They may improve their driver design in the future .

Perhaps elieser will find something to do in your case , i hope so .

[jonjamcam]

Ezze, it's not a problem for me. I'm trying to solve potential issues other users may find . Many keyboards and guitar pedal board these days connect directly to the computer and act like audio interface.

[Ezeer]

ok , i just tried to understand your problem that seemed to be complicated but solved with just some different wiring . I'm not saying that what you want can't be done here . :)

[jonjamcam]

I know more expensive pedals like the gt-10 have internal capabilities to solve this issue. This could be a "Compatibility feature" hidden somewhere for advanced users.....let's see what's Elieser's take.

[elieserdejesus]

I think the Jamtaba current behavior is ambiguous. At moment we have 2 ways to disable audio transmition: 1) xmit button and 2) mute button.

The first option is very clear, but the second is obscure for users. The second approach is anbiguous too, because the mute button has 2 functions: mute and xmit off.
So, we have: 1- the case reported here by Jon 2- The cases where users want use direct monitoring in your audio cards (and mute the inputs to avoid doubled sounds). 3- The problem of ambiguity in mute button.

I have no doubt in this case, the best (but not perfect) solution is transmit muted channels, so the 3 items in the list above are solved.

[jonjamcam]

I think for the regular user, if there's a configuration button that says "click here if you use your pedal board", for example it can be very interesting. Hope it's more general than that, but that's the Jamtaba concept isn't it?

Or click this box if you hear double sound.....something like that......

[elieserdejesus]

Understood! Maybe add a check box (on/off) in the preferences to "Transmit muted channels"?

[jonjamcam]

yea, that will do.......Simple and effective. But we have a problem with the effects and pan. In this case they will have to be disabled :( or we'll have different signals in the room and in the local monitor...

The Midi tracks don't need to be changed I think because this is a audio only issue, so the option should read "Transmit muted audio channels".

Does all this make sense? Too much modifications to solve a tiny issue?

[elieserdejesus]

Too much modifications to solve a tiny issue? Exactly Jon.

I thinked in a "direct monitor" control/button in each input track. But doing this we are showing this control every time in the user interface, and this control will be used rarely. This is not good.

In the first Jamtaba version I put 3 controls in the bottom of main window to show "low detail, normal detail and high detail" in user interface. I think this is a possibility, for example in "high detail" mode the "direct monitor" button appears in each track, but is a future work.

[jonjamcam]

I was thinking in another way to solve this:

I noticed that pedal boards that do not offer complex routing are also 2in 2out simple USB interfaces, so this problem of duplicate sound happens only with one channel.

So, in the audio preferences we can have a check box near the audio interface that says "Local monitor for this interface only". If uncheched nothing happens. If checked then when you select that input in the control section, the channel grays out completely and transmit only without monitoring.

Do you think this could work, guys?

[elieserdejesus]

Jon, this solution is a bit more complicated to program. A button "direct monitoring" in the tracks is more easy to program and more clear to users, but show this button every time don't make sense. So we need a way to change the detail level of user interface. This is more complex, but is usefull for many other things (reduce the controls in midi mode, for example).

[Ezeer]

@jonjamcam i am interested to see your log with the jt.Audio=true when you use your pedal driver . Then we could see what kind of tweak we could do .

[jonjamcam]

@elieserdejesus "A button "direct monitoring" in the tracks is more easy to program and more clear to users"

So direct monitoring will mean in this case:

1.- mute local track audio (so it can be heard directly via the USB hardware) 2.- transmit to the room. 3.- no posibility for vst, volume/pan control.

right?

@Ezeer here's the log tested in win10 x64 jamtaba 2.0.6

log_win10x64.txt

[Ezeer]

Thanks jon . I will follow that : jt.Audio.DEBUG: using WIN32 scpecific stream infos for inputs in 0 jt.Audio.DEBUG: using WIN32 scpecific stream infos for outputs in 0

to see what Jamtaba has in the box . But it is at core PortAudio that we need more info , so perhaps you could use the console program i made for peter to see what we can use on your driver . Peter reported that this console works on x64 too ( just remove .txt extension ) HiJack.exe.txt

[elieserdejesus]

Exactly Jon!

[Ezeer]

My idea would be to create a new kind of LocalTrack based on : LocalTrackView::LocalTrackView(Controller::MainController *mainController, int channelIndex, float initialGain, BaseTrackView::BoostValue boostValue, float initialPan, bool muted) :BaseTrackView( mainController, 1), fxPanel(nullptr), inputNode(nullptr) { init(channelIndex, initialGain, boostValue, initialPan, muted); }

That new local track could inherit from LocalTrackView , but implement specific behavior , without changes in the layout ( mute is donre internally for example etc ....) and created based on a list of audio drivers names stored in a json for example .

The result would be that when the audio driver will be selected in preference and recognized as " special " , the tracks will be " special " too , but keeping the same layout . It's possible also to give a different background color to that tracks to signal that they are " specials ".

[jonjamcam]

@Ezeer

" create a new kind of LocalTrack "

you mean to have audio input, midi input and a custom input as separate type?

"so perhaps you could use the console program "

You mean tweak the driver to disable audio playback in the pedal?

I'm inclined to think this pedals are kind of hard wired.....

If that's possible we'll need to tweak all dirvers of all pedals......I think @elieserdejesus is on the right track on this one....unless I got your idea wrong.

[Ezeer]

It is easier for me to understand the output of the console than an human description of his machine , lol .... that's why i asked you to use the console so that i could see what kind of asio support you have.

The new track type is a feature that could hide to the user the complexity of special routing , without having to change the base tracks ( you create a new type of track instead of adding more buttons in the usual tracks )

If the kind of problem you have is generic ( to different kind of hardwares that could use the same asio driver) we can handle it and provide assorted tracks .

[jonjamcam]

ok I'm running it on win10 x64. now what?

[Ezeer]

copy the screen please and paste here

[jonjamcam]

[Ezeer]

Thanks . Here i see the benefits for u to use your pedal , looking at default low input latency . My idea is for exemple to store the Asio driver name ( here ME-25 ) in a list , that will serve to create special tracks , or directmonitor if it's right to call them like this.

[jonjamcam]

I see, so if the driver is in the list, then it's all automatic? I mean Jamtaba will create a "custom track" not a real time track?

[Ezeer]

what do you call a real time track ?

Inheritance in c++ is a very cool feature that allow you to define objects as parents , then you create objects that inherits the parents behavior but also can modify it or add new ones .

In the case of our special tracks , the parent act like this with addplugin : http://makeitezeeah.esy.es/Jamtaba2/2.0.6/class_local_track_view.html#aca3bd1906700fd367c27b596293ba605

we can define the child of that track to do nothing when the user ask to add a plugin for example . And the child could be LocalDirectMonitorTrack for example . ( or LocalDMTrack for short )

The actual tracks we use are children of a base track , as shown in their constructor : http://makeitezeeah.esy.es/Jamtaba2/2.0.6/class_local_track_view.html#ae445f78dbfac57b281d493458e2ecd0e

the base constructor being : :BaseTrackView( mainController, 1), fxPanel(nullptr), inputNode(nullptr)

We can thus create directmonitor tracks based on BaseTrackView too . It depends of the features we want the parent(s) to already have .

[jonjamcam]

ok, real time track I call the traditional track of jamtaba, capable of vst. It doesn't make sense for this architecture to exist other than in real time.

But a "custom track" can be non-realtime because the USB hardware is doing the job, so it can have a lot of latency and it doesn't make a difference.

I'm using invented names here :D

[Ezeer]

I'm using invented names here :D no problem , ia ma an inventor too . ^^ But we need to share the same references to be able to exchange our ideas .

"But a "custom track" can be non-realtime because the USB hardware is doing the job" USB hardware is more realtime than Asio4All , if you believe that real time is zero latency .

[jonjamcam]

Using non-realtime tracks can open the door for Jamtaba to be used by any computer independent of the ASIO, like the ninjam client. I think Jamtaba has way many great features than the vst capabilities, but these are unavailable for non "pro" users. If you have a latency of say 100 ms jamtaba is pretty unusable and you loose the translate, the chords function, the circular metro, etc....

something to think about

[Ezeer]

i don't get your non-realtime thing , i really need a picture of it . Jamtaba and ninjam are NON-REALTIME JAMMING . We take the time and put it in buffers , that we recompose later . It's ILLUSION of realtime . Realtime is the sound that your pedal produce when you play ( well near of it ... ) ah ah ah .. real time is the sound of your guitar when you play . lol

[jonjamcam]

and realtime is the sound of KONTAKT triggered by my SPD-30 with latency of 2 ms

[Ezeer]

Ah ah ah ! EXACTLY ! ;)

[jonjamcam]

If you use vst, you need some pro drivers and stuff, but if you use a mic and you monitor outside Jamtaba, then the latency can be anything, it's irrelevant and I like that when I'm in my old Pentium.....

[Ezeer]

So if i mix the idea of elieser : " have no doubt in this case, the best (but not perfect) solution is transmit muted channels, so the 3 items in the list above are solved. " and mine ( a new kind of track derived from the original to do the elieser idea in the background )

we perhaps have your solution .

[jonjamcam]

"Custom track" or "direct monitor button"......to be or to be.......:D I like both. The button is easier to implement I think.

[Ezeer]

Probably . But with the kind of track i imagine , you click transmit and that's all . All the stuff is handled into the numeric wiring of jamtaba and hidden to the user . The normal features that can't be used could be hidden or just disabled .

Also , other types of tracks could be created the same way : voice chat track video chat track _ Realtime Jam ( udp based ? )

So we keep the same base design , but implement different features .

[jonjamcam]

interesting....I like it.....create new track---->video chat. I think though there could be a problem with the realtime jam with this approach. It's not compatible with the ninjam rooms.

BTW Realtime JAM! Yea. Jamtaba 1 had realtime rooms, didn't it?

https://github.com/elieserdejesus/JamTaba/issues/183

[jonjamcam]

@elieserdejesus I think this thread got somehow derailed. Just to summarize the ideas related to the specific "direct monitoring implementation" discussed before so it's easier to resume:

1.- In https://github.com/elieserdejesus/JamTaba/issues/182#issuecomment-159679120 you said:

A button "direct monitoring" in the tracks is more easy to program and more clear to users, but show this button every time don't make sense. So we need a way to change the detail level of user interface. This is more complex, but is usefull for many other things (reduce the controls in midi mode, for example).

2.- in https://github.com/elieserdejesus/JamTaba/issues/182#issuecomment-159691872 I posted:

So direct monitoring will mean in this case:

1.- mute local track audio (so it can be heard directly via the USB hardware) 2.- transmit to the room. 3.- no posibility for vst, volume/pan control.

---

Now that some time has passed I think this implementation could benefit VST/AU users too, not only Standalone users. and could be hidden and simple to implement. Just right click on any local channel to display a drop down menu:

This way the interface is no polluted. Do you think it's a good idea?

[elieserdejesus]

@jonjamcam , now the boost buttons are "condensed" in just one button and we have more "head room" to add new buttons without poputing the UI. I think a new "direct monitoring" button is not too bad now.

What you prefer? A new button (like "stereo invert") or a option in the right click menu?

[jonjamcam]

What you prefer? A new button (like "stereo invert") or a option in the right click menu?

I like the idea of a hidden feature because you're right this is not something evryone will use. Mostly people with basic USB interfaces, pedals, keyboards, etc., so I'm inclined to the right click menu.

[jonjamcam]

@NakajimaYusuke proposed the following here https://github.com/elieserdejesus/JamTaba/issues/522#issuecomment-296367293 :

Several ninjamer performances always sound a bit delayed from the metronome. I guess most of the reason of this problem is because they are playing with direct monitoring.

During the NINJAM session, direct monitoring brings monitoring without any latency. This is the advantage compared to "input monitoring". But on the other hand, direct monitoring brings latency between "monitor sound" and "actual transmitted sound".

@elieserdejesus I think this an is important point to consider when implementing the "direct monitoring" feature.

As I understand the problem is quite simple. If you have some latency and you choose direct monitoring you'll be playing that amount of latency behind the metronome. For low latencies this is no problem because <30 ms the ear can't really recognize so little delay as problematic, but if you have 50 or 100 ms of delay now you have a problem.

So what do you think about making this adjustment automatically when users choose direct monitoring in Jamtaba?

As Nakajima said in the same post:

If the direct monitoring delay elimination method is implemented, and all of member uses direct monitoring, a perfectly delay-free session will be realized from anywhere on the Earth.

Of course this implementation does not solve the problem when users use other clients. For that cases we can discuss solutions in the other thread https://github.com/elieserdejesus/JamTaba/issues/522 .

[ghost]

I am a bass player and direct monitoring is essential for me using headphones to not feel disconnected from my bass. I have 13ms round trip latency so it has no effect on others whether I use direct monitoring or not.

I've been using the volume fader to adjust myself out of the mix, hearing just the direct monitor. This appeared to be working fine... when I first started, I was using Mute, which apparently stops xmitting too (but it does not tell the user this - BAD, led to me playing for about 2 weeks without knowing I wasn't being heard).

However I just updated and now even the fader is doing this. This is again not clear to the user and is out of scope for a mixer that is displayed identically to the others.

The ideal fix for me is to fix the mute button to only work locally, the same way it does for every elses channel. the Xmit button is there for when you want to kill sending.

Thanks for considering.

[elieserdejesus]

Hi @Pande !

The ideal fix for me is to fix the mute button to only work locally, the same way it does for every elses channel. the Xmit button is there for when you want to kill sending.

I agree, this is the simple solution. I will fix the mute button behavior (xmit when muted) and keep this issue opened to improve the "direct monitoring" idea in future. The direct monitoring idea contains some complex details like latency auto compensation.

[elieserdejesus]

Hi @Pande , @jonjamcam , @MagnusDrumsTrumpet , @pljones , @DoublebassOO , @qulf !!

A new JamTaba release is available. If possible please help using/testing this pre-release. The change log is in the link too: https://github.com/elieserdejesus/JamTaba/releases/tag/v2.1.4

Thanks in advance!!

[pljones]

Hi,

Overall comment ----| For NINJAM to work, seriously - the only sound you reaching your ears must be through the client. Client can choose to send or not send to server or mix your local input with remote input to despatch to local output. But the only way to local output is through the client when you're ninjamming and the only way from audio source to your ears must be through the client. It kills jams if people do otherwise.|

Anyway.

Very brief session using the 2.1.4-pre build (I've messed up my local dev environment again) - seemed okay. I had video running locally - couldn't find anyone with video remotely at the time, though. Didn't try anything fancy. Wasn't on long. I'll try it again at the weekend.

-- Peter

[ghost]

Thank you very much for reverting the mute button!

To pljones, Sincerely, a 10-15 ms delay which is the most one can expect from roundtrip latency on most sound cards, on my end, is going to do much more to the jam (because it affects my playing) than it will to the overall jam if everyone else hears me 10-15ms late. More delay is experienced in a real room between you and your band mates guitar amplifier, for example.

If we're worried about maybe someone is using WaveOut or something and they are getting a 100-150ms delay... then yeah obviously that's going to affect the jam but that would never last anyway because neither performer nor audience would tolerate it.

[pljones]

If 10-15 ms doesn't affect anyone else, why does affect you? What makes you so special?

[ghost]

Because it affects how you play the instrument, whereas just listening to a delayed ~11ms signal that you have nothing to do with is indiscernible from an on time one.

The same reason applies to the mix fader for yourself, if it's still sending audio at the level of the local mix that's bad because it does not allow the player to make himself louder compared to others locally to better hear note-off technique or how much low end there is in the tone, for example, as a bass player.