Connection problems after configuring input channels while connected to server

[bflamig]

This issue is related to #428 and #547.

Scenario summary: Messing around with client audio input connections while connected to a server seems to cause the server to go haywire, causing the client to sometimes never make it past the "Trying to connect" stage when reconnecting, and to sometimes having the message (from somewhere) showing up in the server logs saying "UPD channel severely degraded." Often this is accompanied by having no public servers show up in the list, regardless of genre. The "trying to connect" problem often survives a client reboot. Only rebooting the server clears this problem up.

Scenario details:

1. I have Windows client computers A and B, and one Linux client C, at home on the same local network.

2. In preparation for a band practice, I started up client A and connected successfully to remote Jamulus server (AWS). No other clients were yet connected.

3. After having connected, I was having trouble getting the input channels configured right in the audio device (a Scarlett 4i4). So I was messing around trying to get the right channels specified and direct monitoring turned off

4. During this messing around, I noticed that somewhere along the way, the client was no longer connected. Either the client disconnected itself (I didn't do it), or the server did.

5. After this, I was no longer able to connect to the AWS server with client A. It would hang on "Trying to connect." However, I was able to connect to other servers (like a few of the public servers) just fine.

6. Tested whether client B could connect to the AWS server. Had no problem doing so.

7. Went back to client A, and after trying multiple times, I decided to check the remote Jamulus server logs. I noticed that client A was indeed showing as being connected via a server log message, but the client was still saying "Trying to connect" on the client. After pushing the disconnect button on the client, the remote server log showed that the client was now disconnected.

8. Step 7 was repeated many times, including shutting down the client A program and restarting, and even rebooting the machine. No matter, client A could not get past the "Trying to connect" message. I noticed that the public server lists were now empty all the time.

9. Gave up and moved to client C, a Linux computer (Rpi4). Connected no problem to the server. But I noticed the input connections weren't set up right and messed around with the Jack settings in qjackctl. Somewhere along the way, Jack gave up and quit and so I shutdown the Jamulus client, got the Jack settings right, and then fired up the client to connect to the server. No dice. Not able to get past the "Trying to connect" message. Server log showed it connecting, and disconnecting after I pushed the client disconnect button.

10. Gave up on client's A and C and went client B (a Windows machine). Decided I would just use it for both my fiddle and my wife's fiddle on the two input channels. It connected just fine to the AWS server.

11. However, I realized I was connected to a Behringer UMC404HD directly, instead of going through the Reaper DAW (via ReaRemote) like I had planned, (so I could add a midi drum channel in Reaper). So I tried reconfiguring the input channels in Jamulus to use Reaper instead of the audio device. This was done while connected to the server.

12. I don't remember the exactly what happened next, other than I had to reconnect client B to the server for some reason. But it wouldn't connect. Same "Trying to connect" message. Same behavior in the remote server logs: it showed the client as having connected, and would show it has having disconnected after I pushed the disconnect button on the client.

13. Other band members had called in during this time and said they could connect to the AWS server, but that the audio was severely garbled.

14. I looked at the remote server performance via htop (which showed nothing unusual), and then the server system log more closely. That's when I noticed a message back earlier in the system log, saying "UDP channel is running in a severely degraded state."

15. At this point, I shut down the Jamulus server and stopped and restarted the actual AWS instance. Fired up the Jamulus server. After this, all clients could now connect with server without issue. After realizing the pattern, I told my band members to be sure to avoid messing around with the device input channels after connecting to the server, but to have everything setup ahead of time.

So there is definitely a pattern: Switching input channels while connected to the server can cause the server to go haywire. One of the symptoms can be the "UDP channel severely degraded" message. However, the server seems to think, via the server log, that the client(s) are connecting okay, and then disconnecting as appropriate. There is some memory, though, of the previous clients that couldn't get past the "Trying to connect" message. They won't be able to get past that message (well randomly they sometimes do). Even rebooting the client does not help. Only rebooting server clears all this up. It could be that just restarting the Jamulus server program would also work. I didn't try that.

[pljones]

It might be helpful if you could post the log from the last working connection until the problem starts appearing, including the "UDP channel severely degraded" message.

Also, if you've time to get it down to a fully reproducible problem that'd be great - but it sounds likely it's going to be tricky.

(As it's the networky stuff, I'll probably not look too closely at this. I still don't understand quite how it works.)

[bflamig]

Will try to do so, but as you say, it can be tricky. And I'm not any kind of network guru either.

I did discover that the server log never really says "disconnected" (maybe it ought to ... hmmm ..) but in my case, I was the only client connected, so after disconnecting, the server log says "stopped" so that's my clue that the server now thinks the client is disconnected. BTW: The latest git builds will say "idling" now, instead of "stopped". That was due to a request by me. But now I can see it would be good for debugging purposes for the server to also specifically state a client has disconnected.

[bflamig]

An update after doing some sleuthing on this issue: It appears the "UDP degrade" thing is probably a red herring. The actual message in the log is something like:

Sep 5 07:51:26 ip-172-XX-XX-XX systemd-resolved[383]: Using degraded feature set (UDP) for DNS server 172.YY.YY.YY.

This morning, this particular message occurred during the boot process of the AWS instance (running Ubuntu 20.04). It also occurred when the Jamulus service program was started. I'm thinking the latter is just a coincidence, because several hours later when the machine was doing nothing in particular (server idling with no Jamulus clients connected), these messages were logged;

Sep 5 11:20:25 ip-172-XX-XX-XX systemd-resolved[383]: Grace period over, resuming full feature set (UDP+EDNS0) for DNS server 172.XX.YY.YY. Sep 5 11:20:25 ip-172-XX-XX-XX systemd-resolved[383]: Using degraded feature set (UDP) for DNS server 172.YY.YY.YY.

So I'm thinking this "UDP degraded" stuff has nothing to do with the problem at hand. I note I can run a Jamulus client just fine under such conditions, and suspect this stuff is "normal" for this particular Ubuntu instance. (I didn't do anything other than defaults when I set up this AWS instance. And then installed Jamulus, nothing else.)

Maybe somebody that knows what any of this means can illuminate. I know very little about networking other than the basics.

There is still the issue of clients changing input channel settings while connected to the server sometimes causing connection problems.

[bflamig]

UPDATE: I haven't been able to replicate this problem while running a server on my local network, with client on same network. I've tried all sorts of nasty stuff, like changing the sample format (from int32 to float 32) in the Reaper ASIO settings, on the fly, while the client and server are connected and running. The server would hiccup on the data for a bit, like you might imagine, but it seems to stay in a good state overall, such that subsequent reconnections by the same client never fail.

Thus, it might be a problem only when connected to a remote server. When I get a chance I'll try firing up a test server for this purpose and see if I can get it to exhibit any weird behavior like that described in this thread. It seems to me like some type of message passing timing problem when a client is disconnected momentarily (for input channel changes) and then reconnected? Something like that?

[bflamig]

UPDATE: More evidence of a problem, which occurred very inconveniently right before band practice the other night. I had set things up on a client (A) and was connected successfully to a remote private server, along with one other client (B) on my same local network. Both were connecting and running just fine. Then a bit later I disconnected (A) briefly to change some routing things in Reaper, (which I use to mix my fiddle and also some midi drums), and then tried to reconnect to the remote server. It wouldn't reconnect.

This problem persisted through (1) rebooting my client machine, and (2) stopping and restarting the Jamulus server on the remote machine, and (3) rebooting the remote server machine itself. All the while, I could still easily connect to any other Jamulus server that I tried. And all the while, Client B on my local network had no problem connecting with the server. The same was true for other band members in different parts of the city. They had no trouble connecting.

So I see three symptoms that always seem to happen:

(1) Connecting two clients that reside on the same local network (behind the ISP) to a remote private server. Now, to the outside world, those clients have the same IP address as far as the server is concerned -- the address provided by the ISP (in this case, COX.) That means there is some other mechanism that distinguishes between the two clients, and that something must be transmitted in the data packets. I haven't been able to figure out what that is. Could someone please enlighten me?

NOTE: It matters not what operating system is being used, in my case, Windows or Linux. I use both. One client is on Windows, the other on Linux. Both can have trouble connecting.

(2) Picture client A (the machine that won't connect) and client B (that has no problem connecting), both on my local network. With client B connected, and then trying to reconnect client A, if I examine client B's settings dialog, and watch its ping and overall delay times, they will start alternating between normal values (16 ms ping, 26 ms overall) and delay of +500 ms, displayed in red. This alternating pattern occurs about every 1/2 second or so and continues until I push the disconnect button on client A. I asked other band members if their settings dialog showed anything unusual. They said no, everything was normal to them, with the exception of garbled audio coming from client B's machine (understandable.) So this symptom of the flashing delay times only occurs locally, on my network. NOTE: It doesn't matter whether client B is connected first or not, client A will not connect to our remote private server. Client A will, however, connect to any other public server. Clearly there is some communication conflict going on locally. However that does not necessarily mean my local router is at fault. It still could be something happening at the Jamulus server -- like perhaps getting to the two client channels confused, such that client A never receives its messages from the server, and instead, those messages are going to client B, causing all sorts of problems. That's my best guess of what is happening.

(2) The remote server shows that Client A is connected, when Client A shows that it is not connected. So there is some message that's getting dropped between the client and server about the connection status. Now the last sentence of the previous paragraph begins to make sense.

(3) Other clients, located elsewhere but connected to the same server, don't see anything unusual.

After trying unsuccessfully to connect with Client A, I gave up and suggested to the band we try a different server. I had already setup another remote server (AWS like the other, located in LA), that I use for experimentation, so we switched to that other server. Client A HAD NO PROBLEM CONNECTING TO THAT OTHER SERVER. But, ironically, now client B WOULD NOT CONNECT. Giving up in semi-amused frustration, I did some reconfiguring and both me and my wife just used a single client so that we could at least have band practice in some form.

After about an hour of practice, I suggested we switch back to the first server. Now, client A HAD NO PROBLEM CONNECTING to the previous server -- the server that it refused to connect to before.

Clearly, there is something going on here that has "memory". Once a client has trouble connecting in the fashion above, it won't connect to the same server until some time passes. The problem persists through reboots of the machines. The only thing that restores the connections is to wait until some amount of time has passed. I have found no other remedy.

NOTE: The server logs show nothing unusual, except for showing that it thinks Client A has connected, when Client A doesn't think it has.

UPDATE: You might wonder if it's a router problem after all? Because I never rebooted it in the above scenario. However, in the past when I've had problems like this, I have rebooted the router, to no avail.

This suggests to me some kind of random timing problem / confusion on the server when handling two clients that have the same public facing IP address.

[WolfganP]

@bflamig most likely a connectivity issue on your side. Which external ports did you forwarded at your router config? Each client in your network you connect to the remote server will use a different port on your side (so each client communications can be identified and routed properly). Just access your router traffic logs and adjust your router forwarding configuration.

[bflamig]

@WolfganP thanks for the reply.

In my local router I only open up ports 22124, 22125, and 22126 so that I can startup a server if I want to (for experimentation) and only for specific machines (which I usually don't have running.) These are Linux machines.

I was under the impression that for clients there's nothing that needs to be done in the router. So what exactly am I supposed to adjust?

Also, this does not explain why, 99% of the time, I can connect both clients to the remote server just fine.

As far as any OS firewalls are concerned, Linux (Raspian Linux) has no firewall, correct?

On Windows I do have a firewall, it allows Jamulus through.

Again, this does not explain why, 99% of the time, I can connect to the remote server just fine. It also doesn't explain why, even in Linux, I can have trouble connecting to the remote server.

I go back to your explanation that ultimately each client uses a different port to connect to the server. Does that mean it first connects using port 22124 on the server (I used the default), and then that port is changed for all later communication with that client?

Who makes the decision on what port is used? Is it in software?

[WolfganP]

I go back to your explanation that ultimately each client uses a different port to connect to the server. Does that mean it first connects using port 22124 on the server (I used the default), and then that port is changed for all later communication with that client?

Who makes the decision on what port is used? Is it in software?

The client app is just interested to connect to server's destination ip:port. If you launch 2 or more clients on the same machine (or home network), the network code assigns the source port as they are available so the traffic can be properly routed to each server:client pair.

See it graphically (jamulus server in .149, 2x clients in .100)

[bflamig]

@WolfganP thanks!

I'll look into wireshark.

I see that port is set in socket.cpp in Jamulus. Now I can do some sleuthing.

[bflamig]

@WolfganP

Well!

I installed Wireshark on my Windows machine (thank you, I didn't know about this program), and started Jamulus on Windows and connected to my remote server, and saw with Wireshark that it was using port 22124 on the server side and port 22134 on the client side for the UDP communication.

Then I disconnected Jamulus, and as a goof decided to try running Jamulus on a Raspberry Pi box, wondering if I'd see anything on Wireshark on Windows. I wasn't expecting to.

It just so happens that when I tried to connect to the remote server on the Raspberry Pi box, I got the dreaded "trying to connect" message. Goodie! That means I just happened to catch the problem in action!

I then looked at Wireshark on the Windows machine. I was amazed to see that during this time, the Wireshark program was seeing message traffic from the Jamulus server going to the Windows machine, server port 22124, client port 22134, even though Jamulus was not running on that machine. So in other words, my Windows machine was getting the traffic meant for the Raspberry Pi!

This confirms my suspicion that the Jamulus traffic is sometimes getting confused.

To recap, I started Jamulus on Windows, connected for a while, and then disconnected. Then I started Jamulus on the Raspberry Pi, tried to connect but couldn't. I could see with Wireshark that traffic meant for the Raspberry Pi was instead going to Windows, even though it was no longer running any Jamulus client. Also, the port numbers were the same both times, (22124 server side, 22134 client side.)

UPDATE: About ten minutes later (the time spent writing this post), I tried running Jamulus on the Raspberry Pi. It now connected to the server just fine, and Wireshark on Windows correctly did not see any Jamulus traffic.

So what does this all mean?

[WolfganP]

@bflamig Hahaha, there was no need for you to use wireshark, it was just what I had at hand to show you what was the effect on connecting two clients to one server, but glad you found another useful tool to explore! :-) If you want to use wireshark for your whole home network, you need to run a remote interface on your router (via tcpdump or rpcapd). What wireshark captures depends on the interface you attach it to.

Back to your issue, what you're seeing seems like a router forwarding issue, I suggest check your rules again, or use triggered instead of forwarding (or simply none and check your router logs to see what is dropped and what not)

[bflamig]

@WolfganP

Well, Wireshark found the problem right away, since I was fortunate to get a connection problem right away! (Pure luck.) And yes, I understand Wireshark is restricted to the machine that it's installed on. That's why I was surprised to see Jamulus packets coming in even though the client was on another machine.

As far as the problem being a port forwarding issue, I fail to see how port forwarding has anything to do with a Jamulus client machine. It's only applicable to a server machine, so that can't be the problem. Also, it can't be an OS firewall problem either, because (1) the Jamulus client on Windows had already proven it could pass through the firewall and communicate to the serer, and (2) in the pure luck scenario given above, the client that failed to connect was on a Linux machine, which, as far as I know, has no OS firewall (does it?), and (3) it wasn't that the packets were blocked by the router or OS firewall, they went to the wrong machine.

Unless I can figure out how this could be a software Jamulus problem, it would seem I need to buy a another router, since it would appear mine is flaky. I assume it's ultimately the router that decides what ports are available and tells the software to use it, (in this case the socket layer of the client machine), and the router then routes all packets with that port to the specified machine, right? And somehow, it's sending those packets to the wrong machine -- a machine that had been assigned that port in the past, but which is now no longer using it.

BTW: The router logs don't go into detail about the ports, only the IP addresses.

[WolfganP]

Well, Wireshark found the problem right away, since I was fortunate to get a connection problem right away! (Pure luck.) And yes, I understand Wireshark is restricted to the machine that it's installed on. That's why I was surprised to see Jamulus packets coming in even though the client was on another machine.

Not the machine where wireshark is installed on, but where the interface under analisis is captured (ie you can put a capture at the router level, and remotely analyze the traffic on any PC on the network, just google "tcpdump or rpcapd on router" and you'll get tons of tutorials).

As far as the problem being a port forwarding issue, I fail to see how port forwarding has anything to do with a Jamulus client machine. It's only applicable to a server machine, so that can't be the problem. Also, it can't be an OS firewall problem either, because (1) the Jamulus client on Windows had already proven it could pass through the firewall and communicate to the serer, and (2) in the pure luck scenario given above, the client that failed to connect was on a Linux machine, which, as far as I know, has no OS firewall (does it?), and (3) it wasn't that the packets were blocked by the router or OS firewall, they went to the wrong machine.

If you have a previous activated forwarding rule, it may kick on and cause issues in your setup, due to forcing the traffic where is not supposed to go. Jamulus' protocol doesn't have any end-to-end validation, so if you inject/divert valid packets the program will process those as intended.

Unless I can figure out how this could be a software Jamulus problem, it would seem I need to buy a another router, since it would appear mine is flaky. I assume it's ultimately the router that decides what ports are available and tells the software to use it, (in this case the socket layer of the client machine), and the router then routes all packets with that port to the specified machine, right? And somehow, it's sending those packets to the wrong machine -- a machine that had been assigned that port in the past, but which is now no longer using it.

TBH, I never had any issues like the ones you're describing (nor read any similar issue report in here), so I don't think is Jamulus related, but some network stuff at your internal LAN/internet bridge that's mixing/blocking packets somehow. It will surely need more detailed troubleshooting for sure.

BTW: The router logs don't go into detail about the ports, only the IP addresses.

Usually any router will work well. I run an alternate firmware on an Asus home router and can't complaint as I have full control on it. But you need to get your hands dirty :-)

[bflamig]

I'm certainly not going to be changing any firmware in the router, if that's what you're getting at. The firmware that's in there is custom to COX, and I'm not about to upset that applecart. If need be, I'll get a new router. The jury is still out on that.

[bflamig]

@WolfganP @corrados

I think I have diagnosed the problem and have a proposed "do this if all else fails" remedy.

I still don't know who's at fault for sure but it's likely my router and/or network switch I have in the path. I ordered a new network switch since it's cheaper than a router and less hassle to install. So I'll try it first. But in the meantime, the following is a proposed immediate remedy for anyone who's having trouble connecting:

1) When a Jamulus client starts, it gets assigned a port that's the default port + 10, and then calls are made to the OS (and ultimately the router?) to see if that port is available. If it is, that becomes the client port the server uses to connect with.

2) In my case, I discovered via Wireshark that it's possible for the UPD packets to be sent to the wrong machine on my home network -- a machine that had been connected to Jamulus earlier but is now supposedly disconnected. Now, since Jamulus always tries default + 10 first, (in the normal case that would be 22134) it tries that port number first. The router (or whoever is responsible) says "Sure! it's available! Go for it!" All the while, for whatever reason, that port really isn't available, and is in fact associated with the other machine. It's highly likely the port used when last connected on that other machine was port 22134. Hence the confusion and conflict.

3) So what to do if this situation occurs? Usually, if you wait some indeterminate amount of time (minutes, hours, next day?) the problem goes away. That's not so nice if you are trying to connect, like, right now for band practice. So I thought, "what if I could tell the Jamulus client to try some other port to start with." After perusing the code, I saw that it looked like I could specify a starting port on the command line, even though the documentation specifically states "for server only." But I could see nothing in the code that prevented it from being used for the client. So I tried it. I specified port 22150 (just for kicks) on the command line. Sure enough, I started up a Jamulus client and it used that as the starting point.

Now, I haven't had the "can't connect" problem in a day or so, so I don't know for sure if this remedy will work should I run into the connection problems discussed, but I'm betting it will.

So what do you think? Is this a viable stop-gap remedy to tell people if they are having problems connecting? If it proves to work, maybe that information should be added to the documentation somewhere?

[bflamig]

@corrados

After thinking about this for a while, I came up with a possible automatic workaround, as given in pull request #625.

[atsampson]

(Catching up on Jamulus messages after a busy couple of weeks...)

@bflamig - this is really interesting. It might help figure out what's going on here if you know a bit about how network address translation (NAT) works with a typical home router setup. Let's suppose that you're using a PC with the local address 192.168.0.10, your router has a single public IP address 4.4.4.4, and your AWS server host is 5.5.5.5.

When Jamulus opens a UDP socket locally, the decision about what port it's going to use is made entirely on the PC; the router isn't involved. So it might decide to use 192.168.0.10 port 22134. It then sends a packet over the network with a source address of 192.168.0.10 port 22134 and a destination address of 5.5.5.5 port 22124. Because this address isn't on your local network, your PC sends it to your router instead.

The router knows that it can't send a packet out to the Internet with a source address of 192.168.0.10 port 22134, because there'd be no way for a reply to come back to that address. So it rewrites the packet -- "source NAT" -- to have the source address 4.4.4.4 port something... and remembers that if it receives a packet in the future from the Internet with the destination address 4.4.4.4 port something, then it must change the destination address to 192.168.0.10 port 22134 so it goes back to the originating PC.

What the something (the NAT-mapped port number) is will depend on the router; it could be the original port number, or something completely random. There's no reason that you couldn't also have a different machine 192.168.0.20 using local port 22134, in which case the router must choose (and remember) different mapped port numbers for the two different flows of packets. So it sounds like one possibility here is that this mechanism is getting confused - the router isn't distinguishing between the two local machines that use the same port. If that's the case, using different local ports for the two machines would indeed be a decent workaround.

(It's also possible that something in Jamulus isn't distinguishing the two connections correctly, although that seems a bit unlikely since as far as I can see all the comparisons check both IP address and port. And there's some extra complexity in that some ISPs also do a layer of "carrier-grade NAT" within their network so multiple customers can share IP addresses... so it might be that something's broken over which you don't have direct control.)

If you're able to reproduce the problem again, it would be interesting to also see packet traces from your server machine, so you can see what the mapped addresses and ports are, and how they correspond to what you're seeing locally. The tcpdump program will give you the same kind of traces as Wireshark from the command line -- something like tcpdump -n not port 22, to show everything except the SSH connection you're coming in on, might do the job?

[bflamig]

@atsampson

Thanks for the info! I'll have to study your comments. I don't know much about networking details, but I'm picking up a lot quickly --- more than I ever wanted to know!

If I catch the problem in action again I'll try using tcpdump on the server side. I hope that's a Linux program because that's what the server is.

[melcon]

I hope that's a Linux program because that's what the server is.

Yes. Assuming it is a Debian/Ubuntu a simple apt-get install tcpdump would take care of installing it on the server. I would add the -w option to the command line so it will record the capture on a file, like this: tcpdump -n not port 22 -w /tmp/packet_capture_01.pcap.

That way you could download the file packet_capture_01.pcap and open it on your local computer using Wireshark or even share with someone else helping you troubleshoot your network.

[bflamig]

@melcon Thanks for the info

[corrados]

As far as I understand this discussion, the problem reported in the initial post was solved by the code provided in #625, i.e. the randomization of the UDP port of the client. This is now already included in the official Jamulus version for a while now. Shouldn't we close this issue then?

[bflamig]

Yes, the randomization seems to be working fine. I haven’t encountered any problems since.

Thanks

Bryan

From: Volker Fischer notifications@github.com Sent: Friday, November 27, 2020 1:47 PM To: corrados/jamulus jamulus@noreply.github.com Cc: Bryan Flamig bryan@robustcircuitdesign.com; Mention mention@noreply.github.com Subject: Re: [corrados/jamulus] Connection problems after configuring input channels while connected to server (#568)

As far as I understand this discussion, the problem reported in the initial post was solved by the code provided in #625 https://github.com/corrados/jamulus/pull/625 , i.e. the randomization of the UDP port of the client. This is now already included in the official Jamulus version for a while now. Shouldn't we close this issue then?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/corrados/jamulus/issues/568#issuecomment-734986090 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AQUBA3Q2GUBVMR5H3XTAA3TSSAF4BANCNFSM4QZRDQOA . https://github.com/notifications/beacon/AQUBA3TCE6GQF35MUKVMC73SSAF4BA5CNFSM4QZRDQOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFPHP62Q.gif

[corrados]

Ok, thanks for your feedback. So, I'll close this Issue now.