Play issue

[curt4207]

When you press the play button the first 3 bar are only audible, when the code loops it plays the next 3 bars after in the loop and so on. [ playButton(this) keeps showing an error on each song] Great program, I would like to make a Warframe related project for my JavaScript / React class not sure what I would want to make this has inspired me to think of something.

[buff0000n]

@curt4207 Huh, I can't reproduce the problem. Can you give me the exact error you're seeing, and what browser you're using?

[curt4207]

My roommate says that it looks like it's lag in the browser. I'm using Edge.

Here's what he says:

I looked at the code on the page, and I think that it's because something is causing the page to freeze, so it's skipping notes. I put a breakpoint on source.play and it was only getting called a few times. I saw something mentioning some logic to handle when the browser drops below 8 FPS, which I bet is the issue.

I think a solution could be to schedule the notes all at once to play at the right time, and let the AudioContext deal with the rest

[buff0000n]

I gotta be honest, I don't test with Edge. I know there's other stuff that doesn't work in Edge, but I can't remember what at the moment.

I think a solution could be to schedule the notes all at once to play at the right time, and let the AudioContext deal with the rest

Unfortunately AudioContext doesn't have a clean way to stop pending sounds from playing without cutting currently playing sounds off in the middle. So I don't schedule sounds until they're a tick away, and when playback is stopped I just let currently playing/scheduled sounds be.

[Tylian]

Kind of old issue but figure I'd throw some input in since I came across it:

I think a solution could be to schedule the notes all at once to play at the right time, and let the AudioContext deal with the rest

This is correct, you schedule everything all at once with the proper delays and just let the browser handle actually playing the sounds at the correct time, because JavaScript's event loop is kind of imprecise.

The way to stop a queued sound is to disconnect the sounce node. Source nodes are meant to be cheap to create an dispose of, so do not worry about making too many.

In my own app that's similar to yours but for a different game, I handle playback the following way:

1. Create the audio context and store it somewhere
2. Create a source note for every single note that needs to be played, calling start and stop with the appropriate delays
3. Save every single note into an array

And when the user requests playback be stopped:

1. call disconnect() on every single saved source node.
2. destroy the audio context.

[buff0000n]

@Tylian

Create a source note for every single note that needs to be played, calling start and stop with the appropriate delays

The music plays in a loop indefinitely. It is impossible to schedule everything in advance. At some interval during playback you need to start scheduling things on the fly.

And when the user requests playback be stopped: call disconnect() on every single saved source node. destroy the audio context.

Unfortunately this will not only cancel pending sounds, but it will also abruptly stop any sounds that have already started playing. I'm trying to avoid that because it's not how the in-game player works. In-game, it stops the sequence but lets any in-progress notes continue to play out. I have not found a reliable way to separate sounds that are actually playing from ones that are still pending.

I do have some ideas on reworking how sequencing and playback works, clearing up glitches and generalizing it so I can maybe reuse it to write a Shawzin player. But it will still work basically the same way to support looping and letting in-progress notes play out.

[Tylian]

The music plays in a loop indefinitely. It is impossible to schedule everything in advance. At some interval during playback you need to start scheduling things on the fly.

A hybrid approach would work here then, start an interval at the same time playback starts, and when the interval fire requeue all the notes with the proper delay, offset by the loop count? Should still leave most of the playback handling to audio device itself.

Something similar to what's suggested here on MDN, specifically scheduler a little bit down from that header.

I have not found a reliable way to separate sounds that are actually playing from ones that are still pending.

The audio context has a currentTime, using that you should be able to calculate which notes have played, which are in the process of being played and which are still pending. Could disconnect all played and pending notes, and then set a timeout to disconnect the playing notes and destroy the context? Sounds pretty tedious but it should accomplish what you want?

---

I'm trying to avoid that because it's not how the in-game player works.

As a side note (and something maybe deserving of a new issue, but just mentioning it here for now), I actually originally came to check this out because I found something that didn't match the in game player. I decided to poke around and see if I could fix it myself and saw that it would be quite hard with the current setup, which led me here because I went to check if anyone filed a related issue already haha.

Specifically with the delta melody line, it seems like the in game player will stop the previous playing note when starting a new one? I noticed it on this song, which I copied from this reddit post.

[buff0000n]

Specifically with the delta melody line, it seems like the in game player will stop the previous playing note when starting a new one?

Huh, how did I not notice that all this time? Now that I look back at it, this has apparently always been the case.

It's actually really easy to make that change, because the Bombast melody and bass do the same thing. There's a trio of mono settings in the instrument pack metadata in song.js that get piped into a flag in audio.js to enable that behavior: SoundBankSource.mono.

The audio context has a currentTime, using that you should be able to calculate which notes have played, which are in the process of being played and which are still pending.

That sets up a race condition where a sound could start after I pull the audio context's current time but before I cancel sources. At best you'll hear a little pop. At worst, if that instrument set is mono then it will stop the previous sound before getting stopped itself, leading to an abrupt stop.

Oh. When stopping playback with a mono instrument the stop would already be scheduled, so I'd have to unschedule that stop event. I'm not even sure that's a thing you can do.

destroy the context

The audio context is never destroyed, because scheduled notes are not the only notes that play. Whenever you click a note to toggle it on, that note plays immediately. This is true even if playback is currently active. Manual and scheduled notes all go through the same handler so they can play nice with each other. That would be pretty hard to do with a mono instrument in the middle of a playback schedule.

You can also click and drag the playback cursor at any time. It plays whatever notes you drag the cursor over until you let it go. If it was in the process of playing then playback is paused while dragging, and resumed when it's dropped.

It's not impossible to combine these behaviors with more in-advance scheduling, but frankly I don't have the inclination right now to rewrite all that code to be even more complicated.

[buff0000n]

Huh, I forgot about the biggest, most obvious problem. You can change the song while it's playing. Having to modify the scheduled notes on the fly to match changes made in the UI would be a freaking nightmare.

[Tylian]

Huh, how did I not notice that all this time? Now that I look back at it, this has apparently always been the case.

It's actually really easy to make that change, because the Bombast melody and bass do the same thing. There's a trio of mono settings in the instrument pack metadata in song.js that get piped into a flag in audio.js to enable that behavior: SoundBankSource.mono.

Yay! I actually didn't see that when skimming the code, so glad that's a lot easier of a fix than what I thought would have to be done.

That sets up a race condition where a sound could start after I pull the audio context's current time but before I cancel sources.

True, which is exactly why the transactional-style api of web audio exists, but unfortunately this very specific edge case of the api is rough to non existant. I might honestly think on this a bit and attempt my own thing because it's honestly bugging me.

Maybe multiple contexts, or reusing the same one but queueing stuff with a start time based on the currentTime of the context? Hm.

The audio context is never destroyed, because scheduled notes are not the only notes that play. [...]

The main reason I'm destroying and recreating the audio context is to reset the playback timing. That said, yeah the approach to playing one shot sounds on top of sequenced stuff is .. not great.

Having to modify the scheduled notes on the fly to match changes made in the UI would be a freaking nightmare.

Is it not the same UI workflow as stopping, loading the score, and starting playback? Because if it's not, yeah I can see that being a nightmare on it's own to implement.

As a side note, I do wanna say thanks for the conversation. I obviously misunderstood the depth of the changes that would be needed to get this idea to work properly and appreciate your patience!

[buff0000n]

reset the playback timing

That's not a big deal, I just get the current context time and make that the new baseline offset when starting playback. Creating audio contexts is kind of a pain because browsers block it unless you're physically inside a user-driven event handler, so I have to create it lazily on the first click of the play button or a note location, as well as load the actual sounds, and then hold onto that thing.

Is it not the same UI workflow as stopping, loading the score, and starting playback?

Nope. That would be a pretty big hiccup every time a note is changed during playback. There isn't really a "loading the score" step, playback works off the same backend data structure that the UI uses. That's how I can do all the wacky stuff without going crazy.

As a side note, I do wanna say thanks for the conversation. I obviously misunderstood the depth of the changes that would be needed to get this idea to work properly and appreciate your patience!

Hey, no problem. I work in software for a living but not with Javascript, and my projects here are even looser than usual, so it's good get called on my crap :)

[buff0000n]

@Tylian Fixed the Delta instrument to be monophonic. I had to make the fade time adjustable to match how the in-game player sounds.

Also, it turns out the Epsilon bass is also monophonic, kind of. Within each tone it's monophonic, so repeating the same tone stops the one before it. But you can still play two different tones simultaneously. Weird. Luckily that was really easy to implement.

Anyway, thanks for the heads up!