Expose VMix features to C# (Example files included)

[walllable]

What can't you do? At the moment, VMix's features aren't exposed to C#. This means that (if/when we can override the default as requested here) we aren't able to change parameters for effects (for example, a low-pass filter on the main output when dying) or route sounds to a particular audio track in a particular graph, and change parameters on those (I believe we can't do this yet, anyway.)

How would you like it to work? Since VMix is pretty much a mixer that can be used to apply different effects and route sounds and such like... Well, a mixer, I think these are the most important things to add:

• Designate which graph and audio track we'd like to route a sound/soundevent to
  • I think we might already be able to do this, based on how we can add sounds in VMix and route them to a specific track, but I'm not sure if this is supposed to do that and it just doesn't work, or if this is just to preview a sound going through a specific track.
  • Being able to route soundstreams to tracks would be great as well!
• Expose control.Input nodes to C# as a float so we can control parameters that we link them up to

What have you tried? Nothing, I'm no programmer, and I haven't seen anyone else discuss this, so I assumed there's no support for it.

Additional context Here's a link to the example file. Everything should be explained pretty clearly in the comment nodes I added I think, but if you're unsure on something feel free to reply of course. Just drag and drop it into your addons folder, open vmix, and open the file in sounds/vmix/ Files: https://files.catbox.moe/fp8xbt.7z Video demonstration: https://files.catbox.moe/cpup4y.mp4

[xezno]

This would be a super cool feature to have, especially for dynamic in-game music!

[walllable]

I made another example file, along with a mockup video. Ideally the input nodes' values would be available to use in C# as floats, and able to be set depending on conditions in-game. The concept video mirrors the available inputs in the vmix file, and (at least roughly) demonstrate how it could be implemented in Instagib: -Preround parameter automation is hardcoded, kick+snare and filter cutoff come in once the three-second mark hits -Hihats begin to come in once you fire, and they begin to open after you and an enemy are nearby for a couple seconds -On death, kick+snare changes to just the kickdrum, bassline's filter cutoff lowers, and the higher octave bassline is muted. Kick+snare come back in once you respawn. -Beat mutes when using grappling hook and in the air, just leaving the hihats and bassline -When another player is killed, hihats and lower-octave bassline mute, and cutoff's lowered

.vmix file and its dependencies (just drop into addons folder): vmixConcept2.zip Video demo of .vmix file: https://files.catbox.moe/28nq06.mp4 Mockup/concept video: https://files.catbox.moe/7l4xvh.mp4

[TheBerkin]

Voicing my support for this too because C# access to VMix would be useful for non-music applications as well!

For instance, I want to have dynamic breathing sounds for my player characters that reflect their current status (e.g. stress, panic, low stamina, etc). This would be the perfect way to implement that.

[MuffinTastic]

Bumping this in hopes more thought is given to it.

To be direct, @garrynewman I think this idea of opening up Source 2's audio engine mostly died in your eyes once you made a GUI for sound stacks and realized what a clusterfuck they are from an end user perspective, or at least it looks that way from the outside. That's entirely valid, and I think the idea you had of having one mega sound stack with a billion parameters that people can choose to use or not use as they wish is a good direction. That being said, I think more freedom should still be had over the final mixing of a game. Dynamic music in games is more important than people realize, just ask your sound guy.

Explanation of how Source 2's audio system works so you know I'm not talking out my ass with all this

Source 2's audio engine as a whole can be summarised as being done in five stages:

1. Sound events - Each sound event has associated sound files, and chooses one sound stack - Each sound event sets parameters made available by its chosen sound stack 2. Sound stacks - From the outside, they're like shaders but for sound / sound events. They are per-sound. - They define parameters as input, to be set by sound events' presets and likely also settable by game code. - Each stack is a node graph in its own right, just like VMix layers. Each node within it uses those either the input parameters for the stack as a whole, or outputs from other nodes. - Final outputs target specific 'tracks'. For example: weapon fire sounds, voiceover, ambient, reverb, music, etc. I believe it is valid to have multiple outputs targeting different tracks, considering the 'reverb' track in Half-Life: Alyx. - Common effects include: - 3D spatialization for sound sources 3. Track mixing - Goes over all the sound stacks and downmixes stacks with matching outputs down to a few tracks. - This is the transition point from per-sound processing to post-processing. - Not configurable as far as I can tell, though I'm not sure what you would configure exactly. 4. VMix - VMix is split up into multiple layers for the convenience of artists, it helps with organization. - Each layer defines parameters as input to be manipulated by game code. - They are not strictly related to tracks in any way. It is not uncommon to have multiple track inputs on the same layer. - Each layer is a node graph. - Layers can send their output to other layers. - Common effects include: - Crossfades - Delays - Distortion - Compression - Etc. 5. Final mixing - Downmixes all of the non-track outputs from each VMix layer into a final stereo composite to be heard by the player. - Has some sort of limiter on it to prevent peaking / clipping. - Not configurable.

Sound events → Tracks → VMix

I think there's a rather simple way to do this in a way that doesn't completely suck: On each sound event, allow users to specify which track(s) to send the output to. Just let it be strings or something, 'cause VMix track input nodes already use string names after all. This can be done ahead of time as part of the .sound file, and it can be overridden in code when you start the sound.

VMix could use some touching up for sure, but as a whole it's already a roughly user-friendly interface, unlike the smelling pile of garbage that is sound stacks. Just skip sound stacks entirely, pretend they don't exist at all, and make the integration between sound events and VMix a bit tighter.

Because it would be a bit hard to know what tracks exist in its current state, you'll have to make some UI for it. My suggestion is to create a new widget that shows available tracks something like this:

• Table Tennis (Game project)
  • paddle_sounds (Track)
    • /path/to/soundevent/asset1.sound (Sound Event)
    • /path/to/soundevent/asset2.sound
• Some other game or map or content project
  • yet_another_track
    • /yet/another/asset.sound
• ...

This way you know which tracks exist, which ones belong to another game, which assets use them by default. VMix really doesn't make use of all of its available space so you could put it down in the bottom left corner right around here.

Short-term VMix integration

• Make the AudioSystem.SetX() methods public
• Start each game with the default/core s&box VMix asset, and add a AudioSystem.OverrideMixGraph() method to allow the user override the current VMix asset being used by the game. This is an explicit voluntary action, if they fuck up their sound that's on them, they should know what they're doing (for now). But to remove some initial confusion, maybe start users out with a preset VMix when they create a new one.

I assume this next bit is easier said than done, hence the short-term stuff.

My ideal form of VMix

Let's face it, having all the layers and effects in the same file sucks. It just sucks! So split it up.

• Do away with layers. Now, there'd be a main vmix asset, then that mix references vsubmixes and gets their outputs with dedicated submix nodes. Double click to go into a submix, just like animgraph. The preview widget should probably show parameters that live in submixes and clearly label them as such.

• The processor nodes now reference vmixefct assets, which are easy to copy around (and maybe could even be published to asset.party? Dunno)

[garrynewman]

Not gonna argue with that. I agree that our audio situation isn't good right now.

I don't think enabling VMix is particularly the way to go about it. I don't think we should invest time in something so complicated and coded in c++ when we could be bringing it all into code.

The only real blocker here is taking a look at it and planning out how it should work. There are a lot of moving parts that need to line up. We need to retain some level of control over the audio - we can't just trust people to do whatever they want.. because we're going to lose menu sounds and volume controls.

This is all a lower priority than everything else right now - but it is on the list of stuff to get done.

[MuffinTastic]

Fair dos yeah, a very slightly restricted version of it done in C# would be really nice. Happy to hear it's on the list ❤