New connections on non-publishing path cause memory leak

Munoon commented 10 months ago

Which version are you using?

v1.3.1

Which operating system are you using?

[x] Linux amd64 standard
[x] Linux amd64 Docker

Describe the issue

RTSP connections on a path, that no one streaming to, cause memory leak issue. In my case, when I was creating reproducer it has increased from 6.238 MiB to 8 MiB in ~5 minutes.

Here is memory usage that I see on server: screenshot-cloud digitalocean com-2023 11 29-10_27_28

Describe how to replicate the issue

Launch server
Constantly read unexsting RTSP stream

To reproduce this issue, I was running mediamtx using Docker:

$ docker run -it -p 8554:8554 -p 9998:9998 -v $PWD/mediamtx.yml:/mediamtx.yml --rm bluenviron/mediamtx:1.3.1

I used this script to create connections:

while true
do
    nohup ffmpeg -i rtsp://127.0.0.1:8554/stream -c copy -f mp4 stream${i}.mp4 &
    sleep 1
done

Configuration File

```yaml ############################################### # Global settings # Settings in this section are applied anywhere. ############################################### # Global settings -> General # Verbosity of the program; available values are "error", "warn", "info", "debug". logLevel: info # Destinations of log messages; available values are "stdout", "file" and "syslog". logDestinations: [stdout] # If "file" is in logDestinations, this is the file which will receive the logs. logFile: mediamtx.log # Timeout of read operations. readTimeout: 10s # Timeout of write operations. writeTimeout: 10s # Size of the queue of outgoing packets. # A higher value allows to increase throughput, a lower value allows to save RAM. writeQueueSize: 512 # Maximum size of outgoing UDP packets. # This can be decreased to avoid fragmentation on networks with a low UDP MTU. udpMaxPayloadSize: 1472 # HTTP URL to perform external authentication. # Every time a user wants to authenticate, the server calls this URL # with the POST method and a body containing: # { # "ip": "ip", # "user": "user", # "password": "password", # "path": "path", # "protocol": "rtsp|rtmp|hls|webrtc", # "id": "id", # "action": "read|publish", # "query": "query" # } # If the response code is 20x, authentication is accepted, otherwise # it is discarded. externalAuthenticationURL: # Enable the HTTP API. api: no # Address of the API listener. apiAddress: 127.0.0.1:9997 # Enable Prometheus-compatible metrics. metrics: no # Address of the metrics listener. metricsAddress: 127.0.0.1:9998 # Enable pprof-compatible endpoint to monitor performances. pprof: no # Address of the pprof listener. pprofAddress: 127.0.0.1:9999 # Command to run when a client connects to the server. # This is terminated with SIGINT when a client disconnects from the server. # The following environment variables are available: # * RTSP_PORT: RTSP server port # * MTX_CONN_TYPE: connection type # * MTX_CONN_ID: connection ID runOnConnect: # Restart the command if it exits. runOnConnectRestart: no # Command to run when a client disconnects from the server. # Environment variables are the same of runOnConnect. runOnDisconnect: ############################################### # Global settings -> RTSP server # Allow publishing and reading streams with the RTSP protocol. rtsp: yes # List of enabled RTSP transport protocols. # UDP is the most performant, but doesn't work when there's a NAT/firewall between # server and clients, and doesn't support encryption. # UDP-multicast allows to save bandwidth when clients are all in the same LAN. # TCP is the most versatile, and does support encryption. # The handshake is always performed with TCP. protocols: [udp, multicast, tcp] # Encrypt handshakes and TCP streams with TLS (RTSPS). # Available values are "no", "strict", "optional". encryption: "no" # Address of the TCP/RTSP listener. This is needed only when encryption is "no" or "optional". rtspAddress: :8554 # Address of the TCP/TLS/RTSPS listener. This is needed only when encryption is "strict" or "optional". rtspsAddress: :8322 # Address of the UDP/RTP listener. This is needed only when "udp" is in protocols. rtpAddress: :8000 # Address of the UDP/RTCP listener. This is needed only when "udp" is in protocols. rtcpAddress: :8001 # IP range of all UDP-multicast listeners. This is needed only when "multicast" is in protocols. multicastIPRange: 224.1.0.0/16 # Port of all UDP-multicast/RTP listeners. This is needed only when "multicast" is in protocols. multicastRTPPort: 8002 # Port of all UDP-multicast/RTCP listeners. This is needed only when "multicast" is in protocols. multicastRTCPPort: 8003 # Path to the server key. This is needed only when encryption is "strict" or "optional". # This can be generated with: # openssl genrsa -out server.key 2048 # openssl req -new -x509 -sha256 -key server.key -out server.crt -days 3650 serverKey: server.key # Path to the server certificate. This is needed only when encryption is "strict" or "optional". serverCert: server.crt # Authentication methods. Available are "basic" and "digest". # "digest" doesn't provide any additional security and is available for compatibility reasons only. authMethods: [basic] ############################################### # Global settings -> RTMP server # Allow publishing and reading streams with the RTMP protocol. rtmp: no # Address of the RTMP listener. This is needed only when encryption is "no" or "optional". rtmpAddress: :1935 # Encrypt connections with TLS (RTMPS). # Available values are "no", "strict", "optional". rtmpEncryption: "no" # Address of the RTMPS listener. This is needed only when encryption is "strict" or "optional". rtmpsAddress: :1936 # Path to the server key. This is needed only when encryption is "strict" or "optional". # This can be generated with: # openssl genrsa -out server.key 2048 # openssl req -new -x509 -sha256 -key server.key -out server.crt -days 3650 rtmpServerKey: server.key # Path to the server certificate. This is needed only when encryption is "strict" or "optional". rtmpServerCert: server.crt ############################################### # Global settings -> HLS server # Allow reading streams with the HLS protocol. hls: no # Address of the HLS listener. hlsAddress: :8888 # Enable TLS/HTTPS on the HLS server. # This is required for Low-Latency HLS. hlsEncryption: no # Path to the server key. This is needed only when encryption is yes. # This can be generated with: # openssl genrsa -out server.key 2048 # openssl req -new -x509 -sha256 -key server.key -out server.crt -days 3650 hlsServerKey: server.key # Path to the server certificate. hlsServerCert: server.crt # By default, HLS is generated only when requested by a user. # This option allows to generate it always, avoiding the delay between request and generation. hlsAlwaysRemux: no # Variant of the HLS protocol to use. Available options are: # * mpegts - uses MPEG-TS segments, for maximum compatibility. # * fmp4 - uses fragmented MP4 segments, more efficient. # * lowLatency - uses Low-Latency HLS. hlsVariant: lowLatency # Number of HLS segments to keep on the server. # Segments allow to seek through the stream. # Their number doesn't influence latency. hlsSegmentCount: 7 # Minimum duration of each segment. # A player usually puts 3 segments in a buffer before reproducing the stream. # The final segment duration is also influenced by the interval between IDR frames, # since the server changes the duration in order to include at least one IDR frame # in each segment. hlsSegmentDuration: 1s # Minimum duration of each part. # A player usually puts 3 parts in a buffer before reproducing the stream. # Parts are used in Low-Latency HLS in place of segments. # Part duration is influenced by the distance between video/audio samples # and is adjusted in order to produce segments with a similar duration. hlsPartDuration: 200ms # Maximum size of each segment. # This prevents RAM exhaustion. hlsSegmentMaxSize: 50M # Value of the Access-Control-Allow-Origin header provided in every HTTP response. # This allows to play the HLS stream from an external website. hlsAllowOrigin: '*' # List of IPs or CIDRs of proxies placed before the HLS server. # If the server receives a request from one of these entries, IP in logs # will be taken from the X-Forwarded-For header. hlsTrustedProxies: [] # Directory in which to save segments, instead of keeping them in the RAM. # This decreases performance, since reading from disk is less performant than # reading from RAM, but allows to save RAM. hlsDirectory: '' ############################################### # Global settings -> WebRTC server # Allow publishing and reading streams with the WebRTC protocol. webrtc: no # Address of the WebRTC HTTP listener. webrtcAddress: :8889 # Enable TLS/HTTPS on the WebRTC server. webrtcEncryption: no # Path to the server key. # This can be generated with: # openssl genrsa -out server.key 2048 # openssl req -new -x509 -sha256 -key server.key -out server.crt -days 3650 webrtcServerKey: server.key # Path to the server certificate. webrtcServerCert: server.crt # Value of the Access-Control-Allow-Origin header provided in every HTTP response. # This allows to play the WebRTC stream from an external website. webrtcAllowOrigin: '*' # List of IPs or CIDRs of proxies placed before the WebRTC server. # If the server receives a request from one of these entries, IP in logs # will be taken from the X-Forwarded-For header. webrtcTrustedProxies: [] # Address of a local UDP listener that will receive connections. # Use a blank string to disable. webrtcLocalUDPAddress: :8189 # Address of a local TCP listener that will receive connections. # This is disabled by default since TCP is less efficient than UDP and # introduces a progressive delay when network is congested. webrtcLocalTCPAddress: '' # WebRTC clients need to know the IP of the server. # Gather IPs from interfaces and send them to clients. webrtcIPsFromInterfaces: yes # List of interfaces whose IPs will be sent to clients. # An empty value means to use all available interfaces. webrtcIPsFromInterfacesList: [] # List of additional hosts or IPs to send to clients. webrtcAdditionalHosts: [] # ICE servers. Needed only when local listeners can't be reached by clients. # STUN servers allows to obtain and share the public IP of the server. # TURN/TURNS servers forces all traffic through them. webrtcICEServers2: [] # - url: stun:stun.l.google.com:19302 # if user is "AUTH_SECRET", then authentication is secret based. # the secret must be inserted into the password field. # username: '' # password: '' ############################################### # Global settings -> SRT server # Allow publishing and reading streams with the SRT protocol. srt: no # Address of the SRT listener. srtAddress: :8890 ############################################### # Default path settings # Settings in "pathDefaults" are applied anywhere, # unless they are overridden in "paths". pathDefaults: ############################################### # Default path settings -> General # Source of the stream. This can be: # * publisher -> the stream is provided by a RTSP, RTMP, WebRTC or SRT client # * rtsp://existing-url -> the stream is pulled from another RTSP server / camera # * rtsps://existing-url -> the stream is pulled from another RTSP server / camera with RTSPS # * rtmp://existing-url -> the stream is pulled from another RTMP server / camera # * rtmps://existing-url -> the stream is pulled from another RTMP server / camera with RTMPS # * http://existing-url/stream.m3u8 -> the stream is pulled from another HLS server / camera # * https://existing-url/stream.m3u8 -> the stream is pulled from another HLS server / camera with HTTPS # * udp://ip:port -> the stream is pulled with UDP, by listening on the specified IP and port # * srt://existing-url -> the stream is pulled from another SRT server / camera # * whep://existing-url -> the stream is pulled from another WebRTC server / camera # * wheps://existing-url -> the stream is pulled from another WebRTC server / camera with HTTPS # * redirect -> the stream is provided by another path or server # * rpiCamera -> the stream is provided by a Raspberry Pi Camera source: publisher # If the source is a URL, and the source certificate is self-signed # or invalid, you can provide the fingerprint of the certificate in order to # validate it anyway. It can be obtained by running: # openssl s_client -connect source_ip:source_port /dev/null | sed -n '/BEGIN/,/END/p' > server.crt # openssl x509 -in server.crt -noout -fingerprint -sha256 | cut -d "=" -f2 | tr -d ':' sourceFingerprint: # If the source is a URL, it will be pulled only when at least # one reader is connected, saving bandwidth. sourceOnDemand: no # If sourceOnDemand is "yes", readers will be put on hold until the source is # ready or until this amount of time has passed. sourceOnDemandStartTimeout: 10s # If sourceOnDemand is "yes", the source will be closed when there are no # readers connected and this amount of time has passed. sourceOnDemandCloseAfter: 10s # Maximum number of readers. Zero means no limit. maxReaders: 0 # SRT encryption passphrase require to read from this path srtReadPassphrase: # If the stream is not available, redirect readers to this path. # It can be can be a relative path (i.e. /otherstream) or an absolute RTSP URL. fallback: ############################################### # Default path settings -> Recording # Record streams to disk. record: no # Path of recording segments. # Extension is added automatically. # Available variables are %path (path name), %Y %m %d %H %M %S %f %s (time in strftime format) recordPath: ./recordings/%path/%Y-%m-%d_%H-%M-%S-%f # Format of recorded segments. # Available formats are "fmp4" (fragmented MP4) and "mpegts" (MPEG-TS). recordFormat: fmp4 # fMP4 segments are concatenation of small MP4 files (parts), each with this duration. # MPEG-TS segments are concatenation of 188-bytes packets, flushed to disk with this period. # When a system failure occurs, the last part gets lost. # Therefore, the part duration is equal to the RPO (recovery point objective). recordPartDuration: 100ms # Minimum duration of each segment. recordSegmentDuration: 1h # Delete segments after this timespan. # Set to 0s to disable automatic deletion. recordDeleteAfter: 24h ############################################### # Default path settings -> Authentication # Username required to publish. # SHA256-hashed values can be inserted with the "sha256:" prefix. publishUser: # Password required to publish. # SHA256-hashed values can be inserted with the "sha256:" prefix. publishPass: # IPs or networks (x.x.x.x/24) allowed to publish. publishIPs: [] # Username required to read. # SHA256-hashed values can be inserted with the "sha256:" prefix. readUser: # password required to read. # SHA256-hashed values can be inserted with the "sha256:" prefix. readPass: # IPs or networks (x.x.x.x/24) allowed to read. readIPs: [] ############################################### # Default path settings -> Publisher source (when source is "publisher") # Allow another client to disconnect the current publisher and publish in its place. overridePublisher: yes # SRT encryption passphrase required to publish to this path srtPublishPassphrase: ############################################### # Default path settings -> RTSP source (when source is a RTSP or a RTSPS URL) # Transport protocol used to pull the stream. available values are "automatic", "udp", "multicast", "tcp". rtspTransport: automatic # Support sources that don't provide server ports or use random server ports. This is a security issue # and must be used only when interacting with sources that require it. rtspAnyPort: no # Range header to send to the source, in order to start streaming from the specified offset. # available values: # * clock: Absolute time # * npt: Normal Play Time # * smpte: SMPTE timestamps relative to the start of the recording rtspRangeType: # Available values: # * clock: UTC ISO 8601 combined date and time string, e.g. 20230812T120000Z # * npt: duration such as "300ms", "1.5m" or "2h45m", valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h" # * smpte: duration such as "300ms", "1.5m" or "2h45m", valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h" rtspRangeStart: ############################################### # Default path settings -> Redirect source (when source is "redirect") # RTSP URL which clients will be redirected to. sourceRedirect: ############################################### # Default path settings -> Raspberry Pi Camera source (when source is "rpiCamera") # ID of the camera rpiCameraCamID: 0 # width of frames rpiCameraWidth: 1920 # height of frames rpiCameraHeight: 1080 # flip horizontally rpiCameraHFlip: false # flip vertically rpiCameraVFlip: false # brightness [-1, 1] rpiCameraBrightness: 0 # contrast [0, 16] rpiCameraContrast: 1 # saturation [0, 16] rpiCameraSaturation: 1 # sharpness [0, 16] rpiCameraSharpness: 1 # exposure mode. # values: normal, short, long, custom rpiCameraExposure: normal # auto-white-balance mode. # values: auto, incandescent, tungsten, fluorescent, indoor, daylight, cloudy, custom rpiCameraAWB: auto # denoise operating mode. # values: off, cdn_off, cdn_fast, cdn_hq rpiCameraDenoise: "off" # fixed shutter speed, in microseconds. rpiCameraShutter: 0 # metering mode of the AEC/AGC algorithm. # values: centre, spot, matrix, custom rpiCameraMetering: centre # fixed gain rpiCameraGain: 0 # EV compensation of the image [-10, 10] rpiCameraEV: 0 # Region of interest, in format x,y,width,height rpiCameraROI: # whether to enable HDR on Raspberry Camera 3. rpiCameraHDR: false # tuning file rpiCameraTuningFile: # sensor mode, in format [width]:[height]:[bit-depth]:[packing] # bit-depth and packing are optional. rpiCameraMode: # frames per second rpiCameraFPS: 30 # period between IDR frames rpiCameraIDRPeriod: 60 # bitrate rpiCameraBitrate: 1000000 # H264 profile rpiCameraProfile: main # H264 level rpiCameraLevel: '4.1' # Autofocus mode # values: auto, manual, continuous rpiCameraAfMode: continuous # Autofocus range # values: normal, macro, full rpiCameraAfRange: normal # Autofocus speed # values: normal, fast rpiCameraAfSpeed: normal # Lens position (for manual autofocus only), will be set to focus to a specific distance # calculated by the following formula: d = 1 / value # Examples: 0 moves the lens to infinity. # 0.5 moves the lens to focus on objects 2m away. # 2 moves the lens to focus on objects 50cm away. rpiCameraLensPosition: 0.0 # Specifies the autofocus window, in the form x,y,width,height where the coordinates # are given as a proportion of the entire image. rpiCameraAfWindow: # enables printing text on each frame. rpiCameraTextOverlayEnable: false # text that is printed on each frame. # format is the one of the strftime() function. rpiCameraTextOverlay: '%Y-%m-%d %H:%M:%S - MediaMTX' ############################################### # Default path settings -> Hooks # Command to run when this path is initialized. # This can be used to publish a stream when the server is launched. # This is terminated with SIGINT when the program closes. # The following environment variables are available: # * MTX_PATH: path name # * RTSP_PORT: RTSP server port # * G1, G2, ...: regular expression groups, if path name is # a regular expression. runOnInit: # Restart the command if it exits. runOnInitRestart: no # Command to run when this path is requested by a reader # and no one is publishing to this path yet. # This can be used to publish a stream on demand. # This is terminated with SIGINT when there are no readers anymore. # The following environment variables are available: # * MTX_PATH: path name # * MTX_QUERY: query parameters (passed by first reader) # * RTSP_PORT: RTSP server port # * G1, G2, ...: regular expression groups, if path name is # a regular expression. runOnDemand: # Restart the command if it exits. runOnDemandRestart: no # Readers will be put on hold until the runOnDemand command starts publishing # or until this amount of time has passed. runOnDemandStartTimeout: 10s # The command will be closed when there are no # readers connected and this amount of time has passed. runOnDemandCloseAfter: 10s # Command to run when there are no readers anymore. # Environment variables are the same of runOnDemand. runOnUnDemand: # Command to run when the stream is ready to be read, whenever it is # published by a client or pulled from a server / camera. # This is terminated with SIGINT when the stream is not ready anymore. # The following environment variables are available: # * MTX_PATH: path name # * MTX_QUERY: query parameters (passed by publisher) # * RTSP_PORT: RTSP server port # * G1, G2, ...: regular expression groups, if path name is # a regular expression. # * MTX_SOURCE_TYPE: source type # * MTX_SOURCE_ID: source ID runOnReady: # Restart the command if it exits. runOnReadyRestart: no # Command to run when the stream is not available anymore. # Environment variables are the same of runOnReady. runOnNotReady: # Command to run when a client starts reading. # This is terminated with SIGINT when a client stops reading. # The following environment variables are available: # * MTX_PATH: path name # * MTX_QUERY: query parameters (passed by reader) # * RTSP_PORT: RTSP server port # * G1, G2, ...: regular expression groups, if path name is # a regular expression. # * MTX_READER_TYPE: reader type # * MTX_READER_ID: reader ID runOnRead: # Restart the command if it exits. runOnReadRestart: no # Command to run when a client stops reading. # Environment variables are the same of runOnRead. runOnUnread: # Command to run when a recording segment is created. # The following environment variables are available: # * MTX_PATH: path name # * RTSP_PORT: RTSP server port # * G1, G2, ...: regular expression groups, if path name is # a regular expression. # * MTX_SEGMENT_PATH: segment file path runOnRecordSegmentCreate: # Command to run when a recording segment is complete. # The following environment variables are available: # * MTX_PATH: path name # * RTSP_PORT: RTSP server port # * G1, G2, ...: regular expression groups, if path name is # a regular expression. # * MTX_SEGMENT_PATH: segment file path runOnRecordSegmentComplete: ############################################### # Path settings # Settings in "paths" are applied to specific paths, and the map key # is the name of the path. # Any setting in "pathDefaults" can be overridden here. # It's possible to use regular expressions by using a tilde as prefix, # for example "~^(test1|test2)$" will match both "test1" and "test2", # for example "~^prefix" will match all paths that start with "prefix". paths: # example: # my_camera: # source: rtsp://my_camera # Settings under path "all_others" are applied to all paths that # do not match another entry. all_others: ```

Did you attach the server logs?

I found interesting, that after ~30 seconds on our production server we start to receive write queue is full log.

Did you attach a network dump?

It looks like mediamtx didn't drop connections. We was using $ netstat -anp | grep :8554 | grep ESTABLISHED | grep mediamtx | wc -l command to count connections and in 2 minutes it increases from 42 to 47, while we have only ~13 publishing connections and most likely only one reading connection.

Munoon commented 10 months ago

After switching from TCP to UDP the problem has disappeared: the memory becomes stable. screenshot-cloud digitalocean com-2023 12 08-09_39_06

aler9 commented 8 months ago

Hello, regarding the memory leak when reading non-existent streams, i ran your script against MediaMTX 1.3.1 with no one publishing, but i couldn't find any trace of it. I enabled pprof in the configuration:

pprof: yes
# Address of the pprof listener.
pprofAddress: 127.0.0.1:9999

and then i monitored both the routine count and the RAM consumption:

docker run --rm -it --network=host golang:1.20 go tool pprof -text http://localhost:9999/debug/pprof/goroutine
docker run --rm -it --network=host golang:1.20 go tool pprof -text http://localhost:9999/debug/pprof/heap

Both reports show that both routine count and RAM consumption stay constant and connections are correctly closed.

Keep also in mind that in Golang, memory is not released immediately but there's a garbage collector that is run periodically and only when RAM consumption exceeds some level, therefore the only way to correctly monitor consumption is by using pprof.

You also mentioned that you are publishing to the server and you switched from TCP to UDP, but both informations seem unrelated from the issue you described, since you said that the issue happens when reading from a non-existing path, and the RTSP handshake always happens with TCP no matter what the transport protocol is.

Is the leak happening under different circumstances than the ones you described?

Munoon commented 8 months ago

Hello. Most likely you are right and memory leaks could also appear when reading existing stream.

Right now we are publishing 12 streams to mediamtx server and have ~100 readers. We have to restart mediamtx each 4 hours to avoid OOM error. screenshot-cloud digitalocean com-2024 01 18-12_04_51

Switching to UDP does resolve the problem for us, but at some point all out streams has crashed and we start receiving packet lost messages in the logs (so we switched back to TCP):

2024/01/09 15:07:50 INF [RTSP] [session a09e2096] created by x.x.x.x:65516
2024/01/09 15:07:50 INF [RTSP] [session a09e2096] is reading from path ‘/xxx’, with UDP, 1 track (M-JPEG)
2024/01/09 15:07:50 WAR [RTSP] [session a09e2096] rtcp: invalid packet version
2024/01/09 15:07:50 WAR [RTSP] [session 9c7f3f1a] 40 RTP packets lost
2024/01/09 15:07:51 WAR [RTSP] [session c0ca7ea1] 46 RTP packets lost
2024/01/09 15:07:51 WAR [RTSP] [session 75e4a1e2] 41 RTP packets lost
2024/01/09 15:07:52 WAR [RTSP] [session c0ca7ea1] 46 RTP packets lost
2024/01/09 15:07:52 WAR [RTSP] [session 9c7f3f1a] 40 RTP packets lost
2024/01/09 15:07:53 WAR [RTSP] [session c0ca7ea1] 46 RTP packets lost
2024/01/09 15:07:54 WAR [RTSP] [session c0ca7ea1] 46 RTP packets lost
2024/01/09 15:07:54 WAR [RTSP] [session 9c7f3f1a] 40 RTP packets lost
2024/01/09 15:07:55 WAR [RTSP] [session 75e4a1e2] 15 RTP packets lost

aler9 commented 8 months ago

@Munoon enable pprof:

# Enable pprof-compatible endpoint to monitor performances.
pprof: yes
# Address of the pprof listener.
pprofAddress: 127.0.0.1:9999

and post here the goroutine and heap reports, that allow to easily identify memory leaks, corresponding to the moment in which memory is almost drawn up. You can generate them with

docker run --rm -it --network=host golang:1.20 go tool pprof -text http://localhost:9999/debug/pprof/goroutine
docker run --rm -it --network=host golang:1.20 go tool pprof -text http://localhost:9999/debug/pprof/heap

Example:

Fetching profile over HTTP from http://localhost:9999/debug/pprof/heap
Saved profile in /root/pprof/pprof.mediamtx.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz
File: mediamtx
Type: inuse_space
Time: Jan 21, 2024 at 3:31pm (UTC)
Showing nodes accounting for 2661.73kB, 100% of 2661.73kB total
      flat  flat%   sum%        cum   cum%
  600.58kB 22.56% 22.56%   600.58kB 22.56%  github.com/go-playground/validator/v10.map.init.4
  522.70kB 19.64% 42.20%   522.70kB 19.64%  regexp/syntax.(*compiler).inst (inline)
  513.56kB 19.29% 61.50%   513.56kB 19.29%  gopkg.in/yaml%2ev2.init.0
  512.88kB 19.27% 80.76%   512.88kB 19.27%  sync.(*Map).LoadOrStore
  512.01kB 19.24%   100%   512.01kB 19.24%  regexp.makeOnePass.func1
         0     0%   100%   512.88kB 19.27%  encoding/gob.Register
         0     0%   100%   512.88kB 19.27%  encoding/gob.RegisterName
         0     0%   100%   512.88kB 19.27%  encoding/gob.init.1
         0     0%   100%   512.88kB 19.27%  encoding/gob.registerBasics
         0     0%   100%   512.88kB 19.27%  encoding/gob.userType
         0     0%   100%   512.88kB 19.27%  encoding/gob.validUserType
         0     0%   100%  1112.60kB 41.80%  github.com/go-playground/validator/v10.init
         0     0%   100%   512.01kB 19.24%  regexp.Compile (inline)
         0     0%   100%   512.01kB 19.24%  regexp.MustCompile
         0     0%   100%   512.01kB 19.24%  regexp.compile
         0     0%   100%   512.01kB 19.24%  regexp.compileOnePass
         0     0%   100%   512.01kB 19.24%  regexp.makeOnePass
         0     0%   100%   522.70kB 19.64%  regexp/syntax.(*compiler).cap (inline)
         0     0%   100%   522.70kB 19.64%  regexp/syntax.(*compiler).compile
         0     0%   100%  2139.04kB 80.36%  runtime.doInit (inline)
         0     0%   100%  2139.04kB 80.36%  runtime.doInit1
         0     0%   100%  2139.04kB 80.36%  runtime.main

File: mediamtx
Type: goroutine
Time: Jan 21, 2024 at 3:31pm (UTC)
Showing nodes accounting for 26, 100% of 26 total
      flat  flat%   sum%        cum   cum%
        24 92.31% 92.31%         24 92.31%  runtime.gopark
         1  3.85% 96.15%          1  3.85%  runtime.goroutineProfileWithLabels
         1  3.85%   100%          1  3.85%  runtime.notetsleepg
         0     0%   100%          1  3.85%  github.com/bluenviron/gortsplib/v4.(*Server).Wait (inline)
         0     0%   100%          1  3.85%  github.com/bluenviron/gortsplib/v4.(*Server).run
         0     0%   100%          1  3.85%  github.com/bluenviron/gortsplib/v4.(*Server).runInner
         0     0%   100%          1  3.85%  github.com/bluenviron/gortsplib/v4.(*serverTCPListener).run
         0     0%   100%          2  7.69%  github.com/bluenviron/gortsplib/v4.(*serverUDPListener).run
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/confwatcher.(*ConfWatcher).run
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/core.(*Core).Wait (inline)
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/core.(*Core).run
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/core.(*pathManager).run
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/protocols/httpserv.(*handlerExitOnPanic).ServeHTTP
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/protocols/httpserv.(*handlerFilterRequests).ServeHTTP
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/protocols/httpserv.(*handlerLogger).ServeHTTP
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/protocols/httpserv.(*handlerServerHeader).ServeHTTP
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/servers/hls.(*Server).run
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/servers/rtmp.(*Server).run
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/servers/rtmp.(*listener).run
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/servers/rtmp.(*listener).runInner
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/servers/rtsp.(*Server).run
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/servers/rtsp.(*Server).run.func1
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/servers/srt.(*Server).run
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/servers/srt.(*listener).run
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/servers/srt.(*listener).runInner
         0     0%   100%          1  3.85%  github.com/bluenviron/mediamtx/internal/servers/webrtc.(*Server).run
         0     0%   100%          1  3.85%  github.com/datarhei/gosrt.(*listener).Accept
         0     0%   100%          1  3.85%  github.com/datarhei/gosrt.(*listener).reader
         0     0%   100%          1  3.85%  github.com/datarhei/gosrt.Listen.func1
         0     0%   100%          1  3.85%  github.com/fsnotify/fsnotify.(*Watcher).readEvents
         0     0%   100%          1  3.85%  github.com/pion/ice/v2.(*UDPMuxDefault).connWorker
         0     0%   100%          5 19.23%  internal/poll.(*FD).Accept
         0     0%   100%          2  7.69%  internal/poll.(*FD).Read
         0     0%   100%          4 15.38%  internal/poll.(*FD).ReadFromInet6
         0     0%   100%         11 42.31%  internal/poll.(*pollDesc).wait
         0     0%   100%         11 42.31%  internal/poll.(*pollDesc).waitRead (inline)
         0     0%   100%         11 42.31%  internal/poll.runtime_pollWait
         0     0%   100%          1  3.85%  main.main
         0     0%   100%          5 19.23%  net.(*TCPListener).Accept
         0     0%   100%          5 19.23%  net.(*TCPListener).accept
         0     0%   100%          4 15.38%  net.(*UDPConn).ReadFrom
         0     0%   100%          4 15.38%  net.(*UDPConn).readFrom
         0     0%   100%          4 15.38%  net.(*UDPConn).readFromUDP
         0     0%   100%          1  3.85%  net.(*conn).Read
         0     0%   100%          1  3.85%  net.(*netFD).Read
         0     0%   100%          5 19.23%  net.(*netFD).accept
         0     0%   100%          4 15.38%  net.(*netFD).readFromInet6
         0     0%   100%          1  3.85%  net/http.(*ServeMux).ServeHTTP
         0     0%   100%          3 11.54%  net/http.(*Server).Serve
         0     0%   100%          1  3.85%  net/http.(*conn).serve
         0     0%   100%          1  3.85%  net/http.(*connReader).backgroundRead
         0     0%   100%          1  3.85%  net/http.HandlerFunc.ServeHTTP
         0     0%   100%          1  3.85%  net/http.serverHandler.ServeHTTP
         0     0%   100%          1  3.85%  net/http/pprof.Index
         0     0%   100%          1  3.85%  net/http/pprof.handler.ServeHTTP
         0     0%   100%          1  3.85%  os.(*File).Read
         0     0%   100%          1  3.85%  os.(*File).read (inline)
         0     0%   100%          1  3.85%  os/signal.loop
         0     0%   100%          1  3.85%  os/signal.signal_recv
         0     0%   100%          1  3.85%  runtime.chanrecv
         0     0%   100%          1  3.85%  runtime.chanrecv1
         0     0%   100%          1  3.85%  runtime.goparkunlock (inline)
         0     0%   100%          1  3.85%  runtime.main
         0     0%   100%         11 42.31%  runtime.netpollblock
         0     0%   100%         11 42.31%  runtime.selectgo
         0     0%   100%          1  3.85%  runtime.semacquire1
         0     0%   100%          1  3.85%  runtime/pprof.(*Profile).WriteTo
         0     0%   100%          1  3.85%  runtime/pprof.runtime_goroutineProfileWithLabels
         0     0%   100%          1  3.85%  runtime/pprof.writeGoroutine
         0     0%   100%          1  3.85%  runtime/pprof.writeRuntimeProfile
         0     0%   100%          1  3.85%  sync.(*WaitGroup).Wait
         0     0%   100%          1  3.85%  sync.runtime_Semacquire

Munoon commented 8 months ago

Here is the dump created right after the launch:

root@streaming-prod:~# go tool pprof -text http://localhost:9999/debug/pprof/goroutine
Fetching profile over HTTP from http://localhost:9999/debug/pprof/goroutine
Saved profile in /root/pprof/pprof.mediamtx.goroutine.002.pb.gz
File: mediamtx
Type: goroutine
Time: Jan 22, 2024 at 2:02pm (UTC)
Showing nodes accounting for 90, 98.90% of 91 total
      flat  flat%   sum%        cum   cum%
        88 96.70% 96.70%         88 96.70%  runtime.gopark
         1  1.10% 97.80%          1  1.10%  runtime.goroutineProfileWithLabels
         1  1.10% 98.90%          1  1.10%  runtime.notetsleepg
         0     0% 98.90%         13 14.29%  bufio.(*Reader).Peek
         0     0% 98.90%         13 14.29%  bufio.(*Reader).fill
         0     0% 98.90%          1  1.10%  github.com/bluenviron/gortsplib/v4.(*Server).Wait (inline)
         0     0% 98.90%          1  1.10%  github.com/bluenviron/gortsplib/v4.(*Server).run
         0     0% 98.90%          1  1.10%  github.com/bluenviron/gortsplib/v4.(*Server).runInner
         0     0% 98.90%         13 14.29%  github.com/bluenviron/gortsplib/v4.(*ServerConn).run
         0     0% 98.90%         13 14.29%  github.com/bluenviron/gortsplib/v4.(*ServerConn).runInner
         0     0% 98.90%         13 14.29%  github.com/bluenviron/gortsplib/v4.(*ServerSession).run
         0     0% 98.90%         13 14.29%  github.com/bluenviron/gortsplib/v4.(*ServerSession).runInner
         0     0% 98.90%         13 14.29%  github.com/bluenviron/gortsplib/v4.(*asyncProcessor).run
         0     0% 98.90%         13 14.29%  github.com/bluenviron/gortsplib/v4.(*serverConnReader).readFuncTCP
         0     0% 98.90%         13 14.29%  github.com/bluenviron/gortsplib/v4.(*serverConnReader).run
         0     0% 98.90%          1  1.10%  github.com/bluenviron/gortsplib/v4.(*serverTCPListener).run
         0     0% 98.90%         13 14.29%  github.com/bluenviron/gortsplib/v4/pkg/bytecounter.(*ByteCounter).Read
         0     0% 98.90%         13 14.29%  github.com/bluenviron/gortsplib/v4/pkg/conn.(*Conn).Read
         0     0% 98.90%         13 14.29%  github.com/bluenviron/gortsplib/v4/pkg/ringbuffer.(*RingBuffer).Pull
         0     0% 98.90%         12 13.19%  github.com/bluenviron/gortsplib/v4/pkg/rtcpreceiver.(*RTCPReceiver).run
         0     0% 98.90%          1  1.10%  github.com/bluenviron/gortsplib/v4/pkg/rtcpsender.(*RTCPSender).run
         0     0% 98.90%          1  1.10%  github.com/bluenviron/mediamtx/internal/confwatcher.(*ConfWatcher).run
         0     0% 98.90%          1  1.10%  github.com/bluenviron/mediamtx/internal/core.(*Core).Wait (inline)
         0     0% 98.90%          1  1.10%  github.com/bluenviron/mediamtx/internal/core.(*Core).run
         0     0% 98.90%         12 13.19%  github.com/bluenviron/mediamtx/internal/core.(*path).run
         0     0% 98.90%         12 13.19%  github.com/bluenviron/mediamtx/internal/core.(*path).runInner
         0     0% 98.90%          1  1.10%  github.com/bluenviron/mediamtx/internal/core.(*pathManager).run
         0     0% 98.90%          1  1.10%  github.com/bluenviron/mediamtx/internal/protocols/httpserv.(*handlerExitOnPanic).ServeHTTP
         0     0% 98.90%          1  1.10%  github.com/bluenviron/mediamtx/internal/protocols/httpserv.(*handlerFilterRequests).ServeHTTP
         0     0% 98.90%          1  1.10%  github.com/bluenviron/mediamtx/internal/protocols/httpserv.(*handlerLogger).ServeHTTP
         0     0% 98.90%          1  1.10%  github.com/bluenviron/mediamtx/internal/protocols/httpserv.(*handlerServerHeader).ServeHTTP
         0     0% 98.90%          1  1.10%  github.com/bluenviron/mediamtx/internal/servers/rtsp.(*Server).run
         0     0% 98.90%          1  1.10%  github.com/bluenviron/mediamtx/internal/servers/rtsp.(*Server).run.func1
         0     0% 98.90%          1  1.10%  github.com/fsnotify/fsnotify.(*Watcher).readEvents
         0     0% 98.90%          3  3.30%  internal/poll.(*FD).Accept
         0     0% 98.90%         14 15.38%  internal/poll.(*FD).Read
         0     0% 98.90%         17 18.68%  internal/poll.(*pollDesc).wait
         0     0% 98.90%         17 18.68%  internal/poll.(*pollDesc).waitRead (inline)
         0     0% 98.90%         17 18.68%  internal/poll.runtime_pollWait
         0     0% 98.90%          1  1.10%  main.main
         0     0% 98.90%          3  3.30%  net.(*TCPListener).Accept
         0     0% 98.90%          3  3.30%  net.(*TCPListener).accept
         0     0% 98.90%         13 14.29%  net.(*conn).Read
         0     0% 98.90%         13 14.29%  net.(*netFD).Read
         0     0% 98.90%          3  3.30%  net.(*netFD).accept
         0     0% 98.90%          1  1.10%  net/http.(*ServeMux).ServeHTTP
         0     0% 98.90%          2  2.20%  net/http.(*Server).Serve
         0     0% 98.90%          1  1.10%  net/http.(*conn).serve
         0     0% 98.90%          1  1.10%  net/http.HandlerFunc.ServeHTTP
         0     0% 98.90%          1  1.10%  net/http.serverHandler.ServeHTTP
         0     0% 98.90%          1  1.10%  net/http/pprof.Index
         0     0% 98.90%          1  1.10%  net/http/pprof.handler.ServeHTTP
         0     0% 98.90%          1  1.10%  os.(*File).Read
         0     0% 98.90%          1  1.10%  os.(*File).read (inline)
         0     0% 98.90%          1  1.10%  os/signal.loop
         0     0% 98.90%          1  1.10%  os/signal.signal_recv
         0     0% 98.90%          1  1.10%  runtime.chanrecv
         0     0% 98.90%          1  1.10%  runtime.chanrecv1
         0     0% 98.90%         14 15.38%  runtime.goparkunlock (inline)
         0     0% 98.90%          1  1.10%  runtime.main
         0     0% 98.90%         17 18.68%  runtime.netpollblock
         0     0% 98.90%         56 61.54%  runtime.selectgo
         0     0% 98.90%          1  1.10%  runtime.semacquire1
         0     0% 98.90%          1  1.10%  runtime/pprof.(*Profile).WriteTo
         0     0% 98.90%          1  1.10%  runtime/pprof.runtime_goroutineProfileWithLabels
         0     0% 98.90%          1  1.10%  runtime/pprof.writeGoroutine
         0     0% 98.90%          1  1.10%  runtime/pprof.writeRuntimeProfile
         0     0% 98.90%         13 14.29%  sync.(*Cond).Wait
         0     0% 98.90%          1  1.10%  sync.(*WaitGroup).Wait
         0     0% 98.90%          1  1.10%  sync.runtime_Semacquire
         0     0% 98.90%         13 14.29%  sync.runtime_notifyListWait
root@streaming-prod:~# go tool pprof -text http://localhost:9999/debug/pprof/heap
Fetching profile over HTTP from http://localhost:9999/debug/pprof/heap
Saved profile in /root/pprof/pprof.mediamtx.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz
File: mediamtx
Type: inuse_space
Time: Jan 22, 2024 at 2:03pm (UTC)
Showing nodes accounting for 3604.52kB, 100% of 3604.52kB total
      flat  flat%   sum%        cum   cum%
  521.05kB 14.46% 14.46%   521.05kB 14.46%  regexp.onePassCopy
  516.01kB 14.32% 28.77%   516.01kB 14.32%  regexp/syntax.appendRange
     515kB 14.29% 43.06%  1028.69kB 28.54%  regexp.makeOnePass.func1
  514.63kB 14.28% 57.34%   514.63kB 14.28%  math/rand.newSource (inline)
  513.69kB 14.25% 71.59%   513.69kB 14.25%  regexp.mergeRuneSets.func2 (inline)
  512.08kB 14.21% 85.79%   512.08kB 14.21%  github.com/gabriel-vasile/mimetype.init
  512.05kB 14.21%   100%   512.05kB 14.21%  regexp/syntax.(*parser).newRegexp (inline)
         0     0%   100%   514.63kB 14.28%  github.com/asticode/go-astikit.init
         0     0%   100%  2065.75kB 57.31%  github.com/go-playground/validator/v10.init
         0     0%   100%   512.05kB 14.21%  github.com/go-playground/validator/v10.init.0
         0     0%   100%   514.63kB 14.28%  math/rand.NewSource (inline)
         0     0%   100%  2577.81kB 71.52%  regexp.Compile (inline)
         0     0%   100%  2577.81kB 71.52%  regexp.MustCompile
         0     0%   100%  2577.81kB 71.52%  regexp.compile
         0     0%   100%  1549.75kB 42.99%  regexp.compileOnePass
         0     0%   100%  1028.69kB 28.54%  regexp.makeOnePass
         0     0%   100%   513.69kB 14.25%  regexp.mergeRuneSets
         0     0%   100%   516.01kB 14.32%  regexp/syntax.(*parser).parseClass
         0     0%   100%   516.01kB 14.32%  regexp/syntax.(*parser).parseUnicodeClass
         0     0%   100%  1028.06kB 28.52%  regexp/syntax.Parse (inline)
         0     0%   100%   516.01kB 14.32%  regexp/syntax.appendTable
         0     0%   100%  1028.06kB 28.52%  regexp/syntax.parse
         0     0%   100%  3604.52kB   100%  runtime.doInit (inline)
         0     0%   100%  3604.52kB   100%  runtime.doInit1
         0     0%   100%  3604.52kB   100%  runtime.main

pprof.mediamtx.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz pprof.mediamtx.goroutine.002.pb.gz

And here is the dump created ~ in 3 hours after the launch:

root@streaming-prod:~# go tool pprof -text http://localhost:9999/debug/pprof/goroutine
Fetching profile over HTTP from http://localhost:9999/debug/pprof/goroutine
Saved profile in /root/pprof/pprof.mediamtx.goroutine.003.pb.gz
File: mediamtx
Type: goroutine
Time: Jan 22, 2024 at 5:01pm (UTC)
Showing nodes accounting for 684, 99.56% of 687 total
Dropped 40 nodes (cum <= 3)
      flat  flat%   sum%        cum   cum%
       684 99.56% 99.56%        684 99.56%  runtime.gopark
         0     0% 99.56%        162 23.58%  bufio.(*Reader).Peek
         0     0% 99.56%        162 23.58%  bufio.(*Reader).fill
         0     0% 99.56%        162 23.58%  github.com/bluenviron/gortsplib/v4.(*ServerConn).run
         0     0% 99.56%        162 23.58%  github.com/bluenviron/gortsplib/v4.(*ServerConn).runInner
         0     0% 99.56%        162 23.58%  github.com/bluenviron/gortsplib/v4.(*ServerSession).run
         0     0% 99.56%        162 23.58%  github.com/bluenviron/gortsplib/v4.(*ServerSession).runInner
         0     0% 99.56%        162 23.58%  github.com/bluenviron/gortsplib/v4.(*asyncProcessor).run
         0     0% 99.56%        162 23.58%  github.com/bluenviron/gortsplib/v4.(*serverConnReader).readFuncTCP
         0     0% 99.56%        162 23.58%  github.com/bluenviron/gortsplib/v4.(*serverConnReader).run
         0     0% 99.56%        150 21.83%  github.com/bluenviron/gortsplib/v4.(*serverSessionMedia).writePacketRTP.func1
         0     0% 99.56%        150 21.83%  github.com/bluenviron/gortsplib/v4.(*serverSessionMedia).writePacketRTPInQueueTCP
         0     0% 99.56%        162 23.58%  github.com/bluenviron/gortsplib/v4/pkg/bytecounter.(*ByteCounter).Read
         0     0% 99.56%        150 21.83%  github.com/bluenviron/gortsplib/v4/pkg/bytecounter.(*ByteCounter).Write
         0     0% 99.56%        162 23.58%  github.com/bluenviron/gortsplib/v4/pkg/conn.(*Conn).Read
         0     0% 99.56%        150 21.83%  github.com/bluenviron/gortsplib/v4/pkg/conn.(*Conn).WriteInterleavedFrame
         0     0% 99.56%         12  1.75%  github.com/bluenviron/gortsplib/v4/pkg/ringbuffer.(*RingBuffer).Pull
         0     0% 99.56%         12  1.75%  github.com/bluenviron/gortsplib/v4/pkg/rtcpreceiver.(*RTCPReceiver).run
         0     0% 99.56%         12  1.75%  github.com/bluenviron/mediamtx/internal/core.(*path).run
         0     0% 99.56%         12  1.75%  github.com/bluenviron/mediamtx/internal/core.(*path).runInner
         0     0% 99.56%          3  0.44%  internal/poll.(*FD).Accept
         0     0% 99.56%        163 23.73%  internal/poll.(*FD).Read
         0     0% 99.56%        150 21.83%  internal/poll.(*FD).Write
         0     0% 99.56%        316 46.00%  internal/poll.(*pollDesc).wait
         0     0% 99.56%        166 24.16%  internal/poll.(*pollDesc).waitRead (inline)
         0     0% 99.56%        150 21.83%  internal/poll.(*pollDesc).waitWrite (inline)
         0     0% 99.56%        316 46.00%  internal/poll.runtime_pollWait
         0     0% 99.56%          3  0.44%  net.(*TCPListener).Accept
         0     0% 99.56%          3  0.44%  net.(*TCPListener).accept
         0     0% 99.56%        162 23.58%  net.(*conn).Read
         0     0% 99.56%        150 21.83%  net.(*conn).Write
         0     0% 99.56%        162 23.58%  net.(*netFD).Read
         0     0% 99.56%        150 21.83%  net.(*netFD).Write
         0     0% 99.56%          3  0.44%  net.(*netFD).accept
         0     0% 99.56%         13  1.89%  runtime.goparkunlock (inline)
         0     0% 99.56%        316 46.00%  runtime.netpollblock
         0     0% 99.56%        354 51.53%  runtime.selectgo
         0     0% 99.56%         12  1.75%  sync.(*Cond).Wait
         0     0% 99.56%         12  1.75%  sync.runtime_notifyListWait
root@streaming-prod:~# go tool pprof -text http://localhost:9999/debug/pprof/heap
Fetching profile over HTTP from http://localhost:9999/debug/pprof/heap
Saved profile in /root/pprof/pprof.mediamtx.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz
File: mediamtx
Type: inuse_space
Time: Jan 22, 2024 at 5:01pm (UTC)
Showing nodes accounting for 69.63MB, 100% of 69.63MB total
      flat  flat%   sum%        cum   cum%
   58.09MB 83.41% 83.41%    63.09MB 90.60%  github.com/bluenviron/gortsplib/v4.(*ServerStream).WritePacketRTPWithNTP
       5MB  7.18% 90.60%        5MB  7.18%  github.com/bluenviron/gortsplib/v4.(*serverSessionMedia).writePacketRTP
    1.01MB  1.45% 92.04%     1.01MB  1.45%  github.com/bluenviron/gortsplib/v4/pkg/ringbuffer.New
    0.52MB  0.74% 92.78%     0.52MB  0.74%  regexp.(*bitState).reset
    0.51MB  0.73% 93.51%     0.51MB  0.73%  regexp.onePassCopy
    0.50MB  0.73% 94.24%     0.50MB  0.73%  github.com/bluenviron/mediamtx/internal/servers/rtsp.(*Server).OnSessionOpen
    0.50MB  0.72% 94.96%     0.50MB  0.72%  regexp/syntax.appendRange
    0.50MB  0.72% 95.69%        1MB  1.44%  regexp.makeOnePass.func1
    0.50MB  0.72% 96.41%     0.50MB  0.72%  math/rand.newSource (inline)
    0.50MB  0.72% 97.13%     0.50MB  0.72%  regexp.mergeRuneSets.func2 (inline)
    0.50MB  0.72% 97.85%     0.50MB  0.72%  github.com/gabriel-vasile/mimetype.init
    0.50MB  0.72% 98.56%     0.50MB  0.72%  regexp/syntax.(*parser).newRegexp (inline)
    0.50MB  0.72% 99.28%     0.50MB  0.72%  time.NewTimer
    0.50MB  0.72%   100%     1.51MB  2.17%  github.com/bluenviron/gortsplib/v4.(*ServerSession).runInner
         0     0%   100%     0.50MB  0.72%  github.com/asticode/go-astikit.init
         0     0%   100%     0.50MB  0.72%  github.com/bluenviron/gortsplib/v4.(*Server).run
         0     0%   100%     0.50MB  0.72%  github.com/bluenviron/gortsplib/v4.(*Server).runInner
         0     0%   100%     1.01MB  1.45%  github.com/bluenviron/gortsplib/v4.(*ServerSession).handleRequestInner
         0     0%   100%     0.50MB  0.72%  github.com/bluenviron/gortsplib/v4.(*ServerSession).initialize
         0     0%   100%     2.01MB  2.89%  github.com/bluenviron/gortsplib/v4.(*ServerSession).run
         0     0%   100%     1.01MB  1.45%  github.com/bluenviron/gortsplib/v4.(*asyncProcessor).allocateBuffer (inline)
         0     0%   100%     0.52MB  0.74%  github.com/bluenviron/gortsplib/v4.(*serverConnReader).readFuncStandard
         0     0%   100%    63.09MB 90.60%  github.com/bluenviron/gortsplib/v4.(*serverConnReader).readFuncTCP
         0     0%   100%    63.60MB 91.34%  github.com/bluenviron/gortsplib/v4.(*serverConnReader).run
         0     0%   100%    63.09MB 90.60%  github.com/bluenviron/gortsplib/v4.(*serverSessionFormat).readRTPTCP
         0     0%   100%    63.09MB 90.60%  github.com/bluenviron/gortsplib/v4.(*serverSessionMedia).readRTPTCPRecord
         0     0%   100%        5MB  7.18%  github.com/bluenviron/gortsplib/v4.(*serverStreamFormat).writePacketRTP
         0     0%   100%     0.50MB  0.72%  github.com/bluenviron/gortsplib/v4.emptyTimer (inline)
         0     0%   100%     0.52MB  0.74%  github.com/bluenviron/gortsplib/v4/pkg/base.(*Request).Unmarshal
         0     0%   100%     0.52MB  0.74%  github.com/bluenviron/gortsplib/v4/pkg/base.ParseURL
         0     0%   100%     0.52MB  0.74%  github.com/bluenviron/gortsplib/v4/pkg/conn.(*Conn).Read
         0     0%   100%     0.52MB  0.74%  github.com/bluenviron/gortsplib/v4/pkg/conn.(*Conn).ReadRequest (inline)
         0     0%   100%    63.09MB 90.60%  github.com/bluenviron/mediamtx/internal/servers/rtsp.(*session).onRecord.func1
         0     0%   100%    63.09MB 90.60%  github.com/bluenviron/mediamtx/internal/stream.(*Stream).WriteRTPPacket
         0     0%   100%    63.09MB 90.60%  github.com/bluenviron/mediamtx/internal/stream.(*streamFormat).writeRTPPacket
         0     0%   100%    63.09MB 90.60%  github.com/bluenviron/mediamtx/internal/stream.(*streamFormat).writeUnitInner
         0     0%   100%     2.02MB  2.90%  github.com/go-playground/validator/v10.init
         0     0%   100%     0.50MB  0.72%  github.com/go-playground/validator/v10.init.0
         0     0%   100%     0.50MB  0.72%  math/rand.NewSource (inline)
         0     0%   100%     0.52MB  0.74%  regexp.(*Regexp).FindStringSubmatch
         0     0%   100%     0.52MB  0.74%  regexp.(*Regexp).backtrack
         0     0%   100%     0.52MB  0.74%  regexp.(*Regexp).doExecute
         0     0%   100%     2.52MB  3.62%  regexp.Compile (inline)
         0     0%   100%     2.52MB  3.62%  regexp.MustCompile
         0     0%   100%     2.52MB  3.62%  regexp.compile
         0     0%   100%     1.51MB  2.17%  regexp.compileOnePass
         0     0%   100%        1MB  1.44%  regexp.makeOnePass
         0     0%   100%     0.50MB  0.72%  regexp.mergeRuneSets
         0     0%   100%     0.50MB  0.72%  regexp/syntax.(*parser).parseClass
         0     0%   100%     0.50MB  0.72%  regexp/syntax.(*parser).parseUnicodeClass
         0     0%   100%        1MB  1.44%  regexp/syntax.Parse (inline)
         0     0%   100%     0.50MB  0.72%  regexp/syntax.appendTable
         0     0%   100%        1MB  1.44%  regexp/syntax.parse
         0     0%   100%     3.52MB  5.06%  runtime.doInit (inline)
         0     0%   100%     3.52MB  5.06%  runtime.doInit1
         0     0%   100%     3.52MB  5.06%  runtime.main

pprof.mediamtx.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz pprof.mediamtx.goroutine.003.pb.gz

aler9 commented 7 months ago

In the pprof you posted there is no sign of any leak, there are 12 publishers (corresponding to the 12 routines that are waiting on ringbuffer.(RingBuffer).Pull) and 150 readers (corresponding to the 150 routines that are waiting on (serverSessionMedia).writePacketRTP.func1).

The RAM consumption is also very low (70 MB).

I tried again to replicate the issue by

creating 40 publishers that are publishing a M-JPEG stream with RTSP/TCP and reconnecting periodically
creating 40 readers that are reading a stream and reconnecting periodically

After some hours, the number of goroutines remains constant and corresponds to the number of publishers and readers:

Showing nodes accounting for 425, 99.53% of 427 total
Dropped 53 nodes (cum <= 2)
      flat  flat%   sum%        cum   cum%
       425 99.53% 99.53%        425 99.53%  runtime.goparki
         0     0% 99.53%         80 18.74%  bufio.(*Reader).Peekn
         0     0% 99.53%         80 18.74%  bufio.(*Reader).fill
         0     0% 99.53%         80 18.74%  github.com/bluenviron/gortsplib/v4.(*ServerConn).run
         0     0% 99.53%         80 18.74%  github.com/bluenviron/gortsplib/v4.(*ServerConn).runInner
         0     0% 99.53%         80 18.74%  github.com/bluenviron/gortsplib/v4.(*ServerSession).rune
         0     0% 99.53%         80 18.74%  github.com/bluenviron/gortsplib/v4.(*ServerSession).runInner
         0     0% 99.53%         80 18.74%  github.com/bluenviron/gortsplib/v4.(*asyncProcessor).runn
         0     0% 99.53%         40  9.37%  github.com/bluenviron/gortsplib/v4.(*serverConnReader).readFuncStandard
         0     0% 99.53%         40  9.37%  github.com/bluenviron/gortsplib/v4.(*serverConnReader).readFuncTCPn
         0     0% 99.53%         80 18.74%  github.com/bluenviron/gortsplib/v4.(*serverConnReader).rund
         0     0% 99.53%          2  0.47%  github.com/bluenviron/gortsplib/v4.(*serverUDPListener).run
         0     0% 99.53%         80 18.74%  github.com/bluenviron/gortsplib/v4/pkg/bytecounter.(*ByteCounter).ReadP
         0     0% 99.53%         80 18.74%  github.com/bluenviron/gortsplib/v4/pkg/conn.(*Conn).Readr
         0     0% 99.53%         80 18.74%  github.com/bluenviron/gortsplib/v4/pkg/ringbuffer.(*RingBuffer).Pull
         0     0% 99.53%         40  9.37%  github.com/bluenviron/gortsplib/v4/pkg/rtcpreceiver.(*RTCPReceiver).run
         0     0% 99.53%         40  9.37%  github.com/bluenviron/mediamtx/internal/core.(*path).run)
         0     0% 99.53%         40  9.37%  github.com/bluenviron/mediamtx/internal/core.(*path).runInnerr
         0     0% 99.53%          5  1.17%  internal/poll.(*FD).Accepts
         0     0% 99.53%         82 19.20%  internal/poll.(*FD).Readd

is there anything else i'm missing?

Munoon commented 6 months ago

I don't think that we really had 150 active readers... However we have some readers, that often reconnects. Maybe mediamtx didn't close those connections for some reasons and still trying to publish message in those channels?

bluenviron / mediamtx