search

Placeholder-Software / Dissonance

Unity Voice Chat Asset
69 stars 5 forks source link

[bug] Photon Bolt Networking instructions need updating #206

Closed SubtleTechnology closed 3 years ago

SubtleTechnology commented 3 years ago

Context

Following the Photon Bolt Networking instructions does not yield basic voice chat functionality

Expected Behavior

Following the instructions for Photon Bolt as described to add it to an existing working Photon Bolt game in early development should yield a very simple always on voice chat.

Actual Behavior

Being very familiar with Photon Bolt, I followed the instructions included for setting up Dissonance with Photon Bolt and they did not yield working voice chat. Simple, open global voice chat did not work - Dissonance debug shows no packets being sent or received other than a tiny initial amount. Dissonance debug output shows it connecting using the Photon Bolt networking but open mic voice chat does not happen.

Workaround

I am very familiar with Photon Bolt, and I was unable to get this to work because I believe it has to do with omissions in the instructions - the setup for the latest version of Dissonance is much more complex than previous versions of Dissonance. I did notice that the microphone was not being selected (no input was selected), so I created a UI similar to what is in the Editor inspector so that I could set it to Microphone. Once that was fixed, speaking did show that the player is talking on a channel but nothing is being sent as far as data - the debug display (showing bytes sent/recv) doesn't show data being sent and received.

Fix

Not obligatory! If you think you know exactly how this problem should be fixed tell us how

Steps to Reproduce

Your Environment

Include as many relevant details about the environment you experienced the bug in

v7.1.0

Unity 2018.4.4

martindevans commented 3 years ago

Did the follow the instructions in the readme included in the bolt demo folder (at Assets/Dissonance/Integrations/PhotonBolt/Demo/README.txt) as well as the online documentation. I just checked and the online docs seem to be a bit out of date/incomplete - they're missing out the steps about creating Events/States required for Dissonance.

If that turns out to be the problem I'll make sure to update the online docs next week to also include those details.

SubtleTechnology commented 3 years ago

Yep I did follow the Readme, i'm familiar with Bolt so I knew it needed some kind of state and recompiled Bolt etc. I think Bolt is setup fine. It may not be a problem with the networking but something basic missing from the steps. The debug shows that it's connected as a Host/Client or Client (depending on who you are) and it shows that it is transmitting in the global channel. I'm not familiar with Dissonance and immediately I noticed that the Basic Microphone Capture had nothing selected as a device, so I added code to make sure that it set properly. It did show the microphone registering speech and like i said it shows trasmitting in the channel in the inspector but no data in the Bolt Comms Network inspector showing anything except a little send/recv for the initial connection. The problem is that I'm not familar with Dissonance, and according to the steps in the documentation it should just work but clearly it's not and there is something missing.. Playback Prefab on the Dissonance Comms is empty - should there be something there? I added the Voice Receipt Trigger and Voice Broadcast Triggers to the scene, and they are both set to Global channel with the "MyAccessToken" token.

SubtleTechnology commented 3 years ago

My scene has DissonanceSetup (Dissonance Comms, Bolt Comms Network, Basic Microphone Capture) DissonanceBroadcastTrigger (Voice Broadcast Trigger seto to Room, Global, with Voice Activation and "MyAccessToken" as token added (with Delete next to it)) DissonanceReceiptTrigger (ChatRoom:Global and "MyAccessToken" as token with Delete next to it) [Trigger is unchecked] also have DissonanceToClient and DissonanceToServer in the Bolt States

martindevans commented 3 years ago

Basic Microphone Capture had nothing selected as a device

This probably isn't a problem, Nothing indicates "use the default microphone device".

Playback Prefab on the Dissonance Comms is empty

This means you don't have a custom playback prefab configured, so again it will use the default one.

token added

Aha! I think this is the issue. triggers with access tokens only activate if you also add the token to the DissonanceComms component itself. Try removing tokens from everything.

SubtleTechnology commented 3 years ago

No change from what I can see. Still only 139 bytes from Server to Clients. Network Status says Connected However, I keep seeing "Sending 73 bytes unreliably to Endpoint XXXX" also I see DissonanceToServer events Where does Dissonance create an AudioSource? I don't see one from it, how does it play the audio? The only AudioSources that I see are mine.

SubtleTechnology commented 3 years ago

[Dissonance:Core] (23:12:54.422) DissonanceComms: Loading default playback prefab [Dissonance:Core] (23:12:54.437) DissonanceComms: Starting Dissonance Voice Comms (7.1.0)

SubtleTechnology commented 3 years ago

Oh, the default playback prefab is not set to DontDestroyOnLoad and I was loading multiple scenes..

martindevans commented 3 years ago

Where does Dissonance create an AudioSource? I don't see one from it, how does it play the audio?

Dissonance instantiates the PlaybackPrefab underneath the GameObject that DissonanceComms is attached to, one per remote speaker.

Oh, the default playback prefab is not set to DontDestroyOnLoad and I was loading multiple scenes

Ah, did marking them as don't destroy resolve the issue?

SubtleTechnology commented 3 years ago

Nope, it's not creating a playback prefab instance. Does it need an event to happen? I'm not using the default player prefab and I didn't see anything in those scripts other than they were setting a playerid somewhere for Dissonance. I do want player positional audio later, but right now was just trying to get plain audio working.

SubtleTechnology commented 3 years ago

It seems like Net_PlayerJoined on DissonanceComms needs to happen, there are other PlayerJoined events all over but it's complicated to follow and I didn't see that being called in the Bolt code. I see EventQueue but thats internal, and SlaveClientCollection has an OnAddedClient.

martindevans commented 3 years ago

Have you tried running the demo scene included in the package? it sounds like you're testing this in your own scene - testing the demo scene would just narrow down the type of error that this can be.

If that doesn't work I'll test out the demo scene myself tmorrow to see if it's just broken (Bolt could have updated since the last time I updated the integration, in which case I'll fix it and send you a new version). If that does work then please tell me some details of your scene setup.

Does it need an event to happen?

Usually Dissonance receives a packet from the Dissonance server telling it that another player has connected, and this triggers allocating a player prefab. Unfortunately it's pretty complex code since it's designed to be flexible enough to fit to a lot of quite different network backends.

SubtleTechnology commented 3 years ago

Yes the demo works. I didn't think there would be a problem there but I did try to debug it. The PlayerJoined events happen in the demo but not in my game, so the question: What are the proper instructions to add Dissonance to a game? Adding the Dissonance components and setting up Bolt correctly isn't all that's needed get you there, something is missing. I thought maybe it was because I don't create the VoiceBroadcastTrigger on the player and it was already in the scene when the game started but that didn't fix it. The interal PlayerJoined event isn't happening when a player connects like it does in the Demo, I looked at all the scripts in the demo folders related to Bolt and none of them do anything that I can see to tell Dissonance a player has connected.

SubtleTechnology commented 3 years ago

So I noticed the VoiceBroadcastTrigger globally and then another one on the player set to Self, and when I set it to Self it said "This entity has no Dissonance player component". So I started looking at the integration code called BoltPlayer and it looks like it wants a DissonancePlayer state on the object. So my only problem is, this is not intuitive since anyone integrating Dissonance will already have their own Bolt Entity and State on their player - and you can't have two States on the same object. So the idea of taking the Bolt Player and adding it to your own game makes no sense. I started looking at that code but it looked like it dealt with player tracking position, and I thought I would get to it later but my guess is that it's all needed and something it does triggers the join event. I will add a Dissonance BoltPlayer with DissonancePlayer state somewhere in the scene when a player joins and see what happens. You also aren't really supposed to parent two entites with each other, but I can see if it will work as a child. I had integrated Dissonance with a Bolt project back in late 2018 very easily and it worked fine, but I remember it not needing all these scripts.

SubtleTechnology commented 3 years ago

I duped the "Bolt Player Prefab" and removed the colliders and models, commented out the code that moves the player but left everything else. It still doesn't trigger the PlayerEntered or whatever events are needed in Dissonance. It just constantly says Sending 73 bytes unreliably to [EndPoint 0.0.0.2:0] and occasionally says: recv event [DissonanceToServer] Raising [DissonanceToServer]

73 bytes isnt even correct, in the demo its all different sizes when you speak. Also it's saying it constantly even if you don't talk. It repeats the same number. Anyways, it says no Peers are connected. Peers (none) So even though networking is connected and seems to be working, something internal didn't register a new player like I described above - the event didn't happen.

SubtleTechnology commented 3 years ago

When I run as server, it sends itself a ClientState and that triggers the PlayerJoined event sequence. When the client connects, it seems like it keeps resending handshake. in BoltCommsNetwork when running as a client, in Update() that checked "Mode" and if it's none it tries to connect as client and it never changed so it keeps queueing up the DissonanceToServer event over and over every few seconds and kept running StartClient() over and over.

martindevans commented 3 years ago

Just to check your basic scene setup, do you have:

This should be all that's required in the scene for the most basic chat set up. Player tracking is not required for basic chat to work (more on that below).

From the other stuff you're saying about packets it sounds as if packets simply aren't getting to where they need to go. Since the demo works it must be something else in the scene. Is there are other special bolt configuration in your scene that may effect things?

Players

Tl;DR: For now you can ignore player tracking entirely.

The player components in Dissonance are purely for position tracking, Dissonance uses these marker components to know where each player is so that it can do positional playback. Here are the docs on general Dissonance positional tracking, bolt specific position tweaks (the intention is that you add one extra field to the state you have already defined for your player entity) and custom position tracking implementations.

The PlayerJoined events happen in the demo but not in my game

This event is triggered by the following process:

  1. Client joins
  2. Sends packet to the server announcing itself
  3. Server replies with session state
  4. Server tells all other clients that the new client has joined
  5. Other clients receive this message and raise the PlayerJoined event

If you look on the inspector for the BoltCommsNetwork on the server you should see Handshake and Client State tick up when a new client joins (indicating everything up to step 3 succeeded). The client that's joining should see Handshake Response and Client State tick up (indicating that step 3 happened). All other client should see Client State tick up (indicating that step 5 happened). Could you check this flow is happening as expected?

occasionally says: recv event [DissonanceToServer]

Is it saying this in the log on the client (you can see the mode the network is in on the BoltCommsNetwork inspector in the Mode field)?

Sending 73 bytes unreliably to [EndPoint 0.0.0.2:0]

This sounds like the handshake request (step 2 in the list above) is sent to the server and never gets there. The client keeps resending the handshake if it doesn't receive a response. If this is the case it would explain why the server doesn't raise the new player event (because it never receives any packets from the new player).

Could you put a breakpoint in Assets/Dissonance/Integrations/PhotonBolt/BoltCommsNetwork.cs Line 133 (SendUnreliableToServer method). And send me the stacktrace, this will hopefully make it apparent which packet this is exactly.

SubtleTechnology commented 3 years ago

Yes everything is setup fine. I found that if I put the components in the last scene that I load that it is working now. Before I had them in the first scene that is around during the main menu along with other things I want kept around. I think the problem is that Dissonance thinks the first time Bolt starts up is the only session it needs and if you shutdown Bolt and make a different connection the Dissonance state is broken. My MainMenu starts Bolt as a Client and displays the Host Browser - the list of hosts available to join. You can either join one or host your own to allow others to join. If you host your own we shutdown Bolt and start it as a Server. It is required for you to startup Bolt as a Client in order to see the list of running hosts. So I would say the majority of Bolt games would do this, the exception would be games that run only dedicated servers and always start as Client and then connect to a host.

SubtleTechnology commented 3 years ago

Here are the objects in my scene now and the scripts and options:

DissonanceSetup object: DissonanceComms BoltCommsNetwork BasicMicrophoneCapture (it doesn't work when scene loads, so I set DissonanceComms.MicrophoneName on Start() and it works now, good thing it has the visual indicator for input otherwise I would never figure this out)

DissonanceReceiptTrigger object: VoiceReceiptTrigger - Chat Room Global DissonanceBroadcastTrigger object: VoiceBroadcastTrigger - ChannelType Room, Chat Room Global

when a player spawns in I create an bolt entity and parent it to my player, similar to the demo: DissonancePlayer object: VoiceBroadcastTrigger - ChannelType Self TriggerVisualizer SpeakerIndicator BoltEntity - IDissonancePlayerState DissonancePlayerTracking BoltPlayerController (with movement/input reading code disabled)

SubtleTechnology commented 3 years ago

I am seeing: [Dissonance:Network] (21:57:24.433) BoltServer: Server received packet 'VoiceData'. This should only ever be received by the client UnityEngine.Debug:LogError(Object) Dissonance.LogMessage:Log() (at Assets/Plugins/Dissonance/Core/Log.cs:67) Dissonance.Logs:SendLogMessage(String, LogLevel) (at Assets/Plugins/Dissonance/Core/Log.cs:96) Dissonance.Log:WriteLog(LogLevel, String) (at Assets/Plugins/Dissonance/Core/Log.cs:179) Dissonance.Log:WriteLogFormat(LogLevel, String, MessageTypes) (at Assets/Plugins/Dissonance/Core/Log.cs:188) Dissonance.Log:Error(String, MessageTypes) (at Assets/Plugins/Dissonance/Core/Log.cs:429) Dissonance.Networking.BaseServer3:NetworkReceivedPacket(BoltPeer, ArraySegment1) (at Assets/Plugins/Dissonance/Core/Networking/BaseServer.cs:263) Dissonance.Integrations.PhotonBolt.BoltServer:PacketReceived(BoltPeer, ArraySegment1) (at Assets/Dissonance/Integrations/PhotonBolt/BoltServer.cs:23) Dissonance.Integrations.PhotonBolt.BoltDissonanceRelay:SendPacketReceived(BoltPeer, ArraySegment1, List`1) (at Assets/Dissonance/Integrations/PhotonBolt/BoltCommsNetwork.cs:135) Dissonance.Integrations.PhotonBolt.BoltDissonanceRelay:StreamDataReceived(BoltConnection, UdpStreamData) (at Assets/Dissonance/Integrations/PhotonBolt/BoltCommsNetwork.cs:119) BoltInternal.GlobalEventListenerBase:StreamDataReceivedInvoke(BoltConnection, UdpStreamData) BoltInternal.BoltCore:Udp_StreamDataReceived(UdpEventStreamDataReceived) BoltInternal.BoltCore:PollNetwork() BoltInternal.BoltCore:Poll() BoltPoll:FixedUpdate()

SubtleTechnology commented 3 years ago

Note that when a player joins there will now be two DissonancePlayer objects (because of replication of the Bolt entity), both will have these scripts. If I need to remove scripts when it gets replicated I can do that in Attach(): DissonancePlayer object: VoiceBroadcastTrigger - ChannelType Self TriggerVisualizer SpeakerIndicator BoltEntity - IDissonancePlayerState DissonancePlayerTracking BoltPlayerController (with movement/input reading code disabled)

If any of these scripts should ONLY be present on the local player and not on remote copies let me know which ones should be removed on remote copies.

SubtleTechnology commented 3 years ago

I just noticed that the VoiceBroadcastTrigger was set to Collider Volume Activation, maybe the Self broadcast is for some kind of testing where you move the player somewhere inside a trigger and they can hear themselves talking via loopback? If so I probably don't need that script, I had just copied the Bolt Player prefab from Demo.

martindevans commented 3 years ago

I set DissonanceComms.MicrophoneName on Start()

If the MicrophoneName is null that should just indicate to Unity that the default mic should be used, so this should work when the scene loads even if it's not set.

BoltServer: Server received packet 'VoiceData'.

I think this probably confirms my theory before that packets are being sent to the wrong place, and so the client is never joining the session becasue the handshake packets are getting lost. Something that slightly confuses me is that the line numbers in this stack trace seem to be slightly off - could you double check that you're using the latest version of the Dissonance+Bolt integration.

I've just gone through and reviewed the code related to sending and receiving packets in the bolt integration and I can't see a way that this should happen, it's possible that this could be a bug in bolt with unreliable data streams (we had a similar bug recently which required rewriting the way unreliable packets are sent to work around another bug).

It sounds like you probably know bolt better than I do, so if you want to double check my work here's a brief overview of how it works:

Note that when a player joins there will now be two DissonancePlayer objects

If I understand what you'ree describing correctly, that's fine. The way player tracking works is every specific client should have a tracker for every player object in the game. Player tracking doesn't send any position data over the network, it purely tells the local instance of Dissonance where each player is. Replacating those scripts is a pretty good way to achieve that.

I just noticed that the VoiceBroadcastTrigger was set to Collider Volume Activation, maybe the Self broadcast is ...

Similar to player tracking triggers are a purely local thing. If you want voice to be sent from me to you when my player object stands near your player object there needs to be a trigger attached to your GameObject on my machine.

Collider Volume Activation means that the broadcast trigger activates when a collider activates, you can use this for proximity chat (docs: https://dissonance.readthedocs.io/en/latest/Tutorials/Proximity-Chat/index.html). e.g. in the demo scene when the players walk within the circle of the other player voice is sent.

Self broadcast means that voice is sent directly to the player which this GameObject represents. This just means that the broadcast trigger looks at the sibling player tracker component and sends voice directly to that ID. By combining these two things you can easily do proximity activated direct voice chat.

SubtleTechnology commented 3 years ago

The only problem that I see right now is that if I have Dissonance in my Main Menu scene, the one that starts Bolt as a Client so that it can display the list of Servers running, then once the game is started I have the problem where clients connecting to the game can't register with Dissonance. By putting the Dissonance objects in the actual destination scene that is loaded by Bolt, it works fine.

I think something about the state within Dissonance is seeing the initial Bolt startup but not that it shutdown nor the second startup. It may just be the Bolt StartClient and then Bolt ShutDown and then Bolt StartServer that is causing confusion and a problem with Dissonance internal state - while Dissonance was kept around and not shutdown or destroyed.

I can just leave Dissonance in the actual game scene, and it's good that I have a solution but i'm wondering what Dissonance is doing or keeping around that the shutdown and second startup of Bolt won't let clients register within Dissonance. It seems like something internal to Dissonance since I see the handshake being received by Dissonance's code but I get lost once it adds the Event and goes to process the handshake. Whatever is happening, it's not then ending up with a PlayerJoined event being created.

At first I thought maybe the server wasn't hitting the OnEvent() callback for the DissonanceToServer for the handshake, but it was receiving it fine, it's just that I'm not able to trace down through that call to figure out why the server code ignores the client's handshake and never registers then and sends a response.

A possible test with the Demo would be to modify the code to call Bolt StartClient, and then Bolt Shutdown, and then Bolt StartServer - then have the other copy try to connect as Client.

Another fix might be if I knew what to call to tell Dissonance to Shutdown cleanly, so that it sees the second StartServer.

One possibility might be that Dissonance sees the Bolt StartClient() and marks itself as a Client, and then the Bolt StartServer is ignored or Dissonance doesn't like that unless you ask Dissonance to Shutdown somehow so that its in a clean state.

I don't really need Dissonance to be in the top level scene, since no connection is established yet anyways so I don't know how important this is but at least now I know that I should only let Dissonance objects be created in the final loaded game scene after a connection has been established and it's certain whether we are a client or server.

SubtleTechnology commented 3 years ago

OK Thank you for more details, I read the documentation on positional and proximity and not sure that I get it, I will try to read again and your notes here.

Positional audio didn't see to work, I would think that the AudioSource needs to move with the player. The PlaybackPrefab is created under DissonanceSetup and never moves for me, although I have positional audio checkbox enabled. I didn't compare with the Demo yet but I don't think that I did anything wrong there. I would think that just attaching the PlaybackPrefab to the player instead of having DissonanceSetup as the parent would fix it, but sure you could also just keep moving the PlaybackPrefab instance around with the player although i'm not seeing that happening. If you do something else like playing audio at an X,Y,Z it wasn't doing that - the audio came from the audiosource. I will remove the Broadcast Self on the players and some other things to remove anything that might be causing problems.

martindevans commented 3 years ago

Reharding the issue with scene transitions, you can fine the main code that controls what state Dissonance is in here. As you can see it checks every frame that Dissonance is running in the same mode as Bolt, as soon as there's a difference Dissonance should switch mode. You can force Dissonance out of it's current mode by calling Stop on the BoltCommsNetwork object, that loop will kick in and start Dissonance back up again next frame. That shouldn't be necessary though if that update loop is keeping things in sync - could you check exactly what it does on your scene transition?

I read the documentation on positional and proximity and not sure that I get it

The overall concept of it is quite simple. The player tracker script tells Dissonance where each player is (using DissonanceComms.LocalPlayerName as the unique identifier for each player). Dissonance then moves the playback objects (spawned under the DissonanceComms setup) to the correct location in the scene.

There are a few ways this can seem not to work:

martindevans commented 3 years ago

I'll close this issue since the discussion seems to have tailed off, feel free to re-open or to drop into Discord chat if you still need help.