Stream - very high packet loss rate with teamspeak3

[pgabrys94]

Checklist

• Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?
  • Yes
• Are you sure you're not using someone else's docker image?
  • Yes
• Have you searched for similar issues (both open and closed)?
  • Yes

Describe the bug

Really high packet loss rate (around 80%) when using stream option to pass traffic on 9987:9987/udp. No issues when skipping proxy (direct from router to ts3 server)

Nginx Proxy Manager Version

2.11.1

To Reproduce Steps to reproduce the behavior:

1. Set ports in docker-compose (9987:9987/udp, 30033:30033/tcp, 41144:41144/tcp) and run it,
2. Port forward on router to proxy vm
3. Set streams on npm webui
4. Use stereo-mix to constantly stream music/movie/dialogue/anything in teamspeak3 (or just try to communicate with somebody)
5. Try to listen to it - u will hear approx. 8sec flawless sound, then chirping and burping for 3 sec, and again and again...
6. Look at client connection info and notice huge packet loss ratio.
7. Occasionally, experience full reconnect to server.

Expected behavior

No packet loss

Screenshots

Operating System

Both proxy and teamspeak are running on PVE

Additional context

-using pve firewall -ping seems ok, around 33ms from client to server by WAN connection

[pgabrys94]

Why did you put the TeamSpeak server behind NPM? This network flow feels pointless to me. Just expose the TeamSpeak server directly.

Why do you bother with answer if it doesn't bring anything up? NPM offers this functionality so i used it. Why? Maybe as another security layer, maybe i just want to, maybe i'm trying out things. Point is: it's not working how it is supposed to work.

[bluekitedreamer]

This may be due to a couple of things.

1. Nginx requires a little more setup for UDP traffic, otherwise it treats most connections similar to a TCP connection. This is where some custom configuration is needed[1]
2. NginxProxyManager's default stream template might not be sufficient enough for the current setup [2], this setup is a tiny bit more complicated than a typical tcp connection reverse proxy and will require some custom configuration. Unlink the proxy host option the stream option does not look like it allows for custom configuration so you're going to have to get down and dirty for this one

[1] I found this article which basically states everything I wanted to say in this regard. I heavily use nginx to reverse proxy streams of all types but have never documented any of the strangeness I've had to overcome. The author here describes things in decent detail for UDP connections.

[2] This is the stream template which does not have any of the nginx directives/configuration options mentioned in the link in [[1]]

Basic Example to Demonstrate above issues

My basic example exemplifies my above comments, I created a dummy stream from incoming port 8888 to localhost port 8887 to show the config that's generated.

# ------------------------------------------------------------
# 8888 TCP: 0 UDP: 1
# ------------------------------------------------------------

server {
  listen 8888 udp;
listen [::]:8888 udp;

  proxy_pass 127.0.0.1:8887;

  # Custom
  include /data/nginx/custom/server_stream[.]conf;**Fixing things in the future** 
1. There needs to be an enhancement for NPM to allow custom config for streams

  include /data/nginx/custom/server_stream_udp[.]conf;
}

Take note of the last line here. While custom configuration is not currently available through the NPM interface, it is available by custom configuration file that will not break NPM's functionality.

Fixing things in the future

There needs to be an enhancement for NPM to allow custom config for streams. I filed enhancement here

[bluekitedreamer]

@pgabrys94 Let me know if above explanation is enough to point you in the right direction. If you read the article I linked in [[1]] it should elaborate on some of the timeouts and connection issues you're hitting here.

If that stuff does not directly fix the problem it should give you enough insight into how to debug, and some of the discrepancies involved. The other thing I might mention is tcpdump the container itself as well as the host of the container engine

[pgabrys94]

@pgabrys94 Let me know if above explanation is enough to point you in the right direction. If you read the article I linked in [[1]] it should elaborate on some of the timeouts and connection issues you're hitting here.

If that stuff does not directly fix the problem it should give you enough insight into how to debug, and some of the discrepancies involved. The other thing I might mention is tcpdump the container itself as well as the host of the container engine

Thanks for your response @bluekitedreamer, this article you have pointed allowed me to fix my issue. However, adding custom configuration in separate file broke proxy completely (yet there is a chance i did something wrong).

My solution was to append "reuseport" flag directly to "listen" directive in data/nginx/stream/*.conf:

`server { listen 9987 udp reuseport;

proxy_pass my.ts3.local:9987;

include /data/nginx/custom/server_Stream[.]conf; include /data/nginx/custom/server_Stream_udp[.]conf;`

Although i do know it is not persistent, since changing anything by WebUI will replace config content. I wasn't able to find how to just append "reuseport" at the end of a listen directive, and copying whole config content into server_stream_udp.conf (without includes) with added "reuseport" kept crashing my proxy.

[bluekitedreamer]

@pgabrys94 Awesome, glad to hear you got it working!

Although i do know it is not persistent, since changing anything by WebUI will replace config content. I wasn't able to find how to just append "reuseport" at the end of a listen directive, and copying whole config content into server_stream_udp.conf (without includes) with added "reuseport" kept crashing my proxy.

Okay that's important that you know that, any GUI changes will override this (I think). I haven't torn apart the innards of the stream templating code in this project so I can't say for sure.

The main nginx.conf looks like this for stream config file reference:

stream {
    # Files generated by NPM
    include /data/nginx/stream/*.conf;

    # Custom
    include /data/nginx/custom/stream[.]conf;
}

include /data/nginx/custom/stream[.]conf; This is the important piece, instead of important everything using * , the config is looking for a specific filename.

What did you name your custom stream file and which directory did you place it in?

/data/nginx/custom/stream.conf

It's intention is to put server code blocks in that file. In this case you could copy the generated code block plus your addition of the reuse line into this file. Keep in mind this would only exist outside NPM. It essentially allows you to run custom server blocks of which cannot be done inside NPM, basically your use case.

You could also place it here include /data/nginx/custom/server_Stream[.]conf; as noted in the config you pasted.

I misspoke in my other bug creation, NPM is not allowing you to extend the server block like I thought it was.

[bluekitedreamer]

I missed this point in your comment

and copying whole config content into server_stream_udp.conf

The file you used is named server_stream_udp.conf or server_Stream_udp.conf? The capital S in the second word Stream is important here.