Closed bpgross closed 4 years ago
Ullaakut
commented
5 years ago Hi @bpgross !
Mh, this shouldn't happen. I've been using up to 20 streams simultaneously on one server for weeks in a row without any issues 🤔 But I might have recently introduced a regression somehow so this is worrying.
What makes you think it's related to the jack server? Are you using a custom pipeline?
Please give more information if you can, I'll make it my priority to help you on this ASAP 🙏🏽
bpgross
commented
5 years ago Thanks for your quick response to this issue.
With respect to the pipeline, I don't "think" I'm using a custom one, here is a typical docker run statement:
docker run -d --name="rtsp-turnstile" -h="rtsp-turnstile" -e INPUT=/tmp/video.wmv \ -e RTSP_RESOLUTION=320x240 -e RTSP_FRAMERATE=30 \ -v /Users/bpgross/code/../TurnStile.wmv:/tmp/video.wmv -p 8554:8554 ullaakut/rtspatt
I am specifying the resolution and pipeline as for my use it is important that we have the stream match the video file, and I could not find another way to do that. My other invocations are similar except that they have a different external port (eg. 8556:8554, a different frame rate (most are 15 fps) or a different resolution. Of course the video file is different - some are m4v files and all file extensions are changed as a result.
I picked on the jack server as docker logs had this for one of the streams that did not work:
Launching stream with the following pipeline: ( appsrc name=mysrc ! decodebin ! videoscale ! video/x-raw,width=320,height=240 ! videorate ! video/x-raw,framerate=30/1 ! capsfilter ! queue ! x264enc speed-preset=superfast ! rtph264pay name=pay0 pt=96 ) Stream ready at rtsp://0.0.0.0:8554/live.sdp ALSA lib confmisc.c:767:(parse_card) cannot find card '0' ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_card_driver returned error: No such file or directory ALSA lib confmisc.c:392:(snd_func_concat) error evaluating strings ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_concat returned error: No such file or directory ALSA lib confmisc.c:1246:(snd_func_refer) error evaluating name ALSA lib conf.c:4528:(_snd_config_evaluate) function snd_func_refer returned error: No such file or directory ALSA lib conf.c:5007:(snd_config_expand) Evaluate error: No such file or directory ALSA lib pcm.c:2495:(snd_pcm_open_noupdate) Unknown PCM default AL lib: (EE) ALCplaybackAlsa_open: Could not open playback device 'default': No such file or directory Cannot connect to server socket err = No such file or directory Cannot connect to server request channel jack server is not running or cannot be started JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
If there is additional information I can provide please let me know.
Thanks,
Brent.
Ullaakut
commented
5 years ago Alright so my guess is that it's due to the sound track of the videos you are streaming, I've always used video without sound so I'll have to experiment with that to try to reproduce (and fix) this issue.
Might also be a problem of not having enough resources available for streaming multiple videos simultaneously, since with the default pipeline everything is re-encoded IIRC, this makes RTSPATT use a significant amount of CPU, and I'm not sure what happens when the CPU isn't able to keep up? The streams might crash, or they might just be slowed down. I'll have to verify if that couldn't also be the source of your problem.
I'll let you know if I find a solution, and in the meantime, sorry for the inconvenience 🙏🏽
bpgross
commented
5 years ago Thanks again for your quick response. At this point in time even though the videos have sound, I'm really only interested in the video portion, so I have been playing around with using a custom pipeline:
-e GST_PIPELINE="filesrc location=/tmp/video.wmv ! decodebin ! videoconvert ! x264enc speed-preset=superfast ! rtph264pay name=pay0 pt=96 " This works for a short period of time (with multiple streams), then all but one of the streams stops and I see this in the log:
(rtspatt:1): GStreamer-CRITICAL **: 15:50:48.728: gst_util_set_object_arg: assertion 'G_IS_OBJECT (object)' failed
(rtspatt:1): CRITICAL : 15:50:48.729: gst_app_src_get_max_bytes: assertion 'GST_IS_APP_SRC (appsrc)' failed
(rtspatt:1): GLib-GObject-CRITICAL **: 15:50:48.729: g_object_set: assertion 'G_IS_OBJECT (object)' failed
(rtspatt:1): GLib-GObject-WARNING **: 15:50:48.729: invalid (NULL) pointer instance
(rtspatt:1): GLib-GObject-CRITICAL **: 15:50:48.729: g_signal_connect_data: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed
I'm not sure if it is due to my pipeline or some other issue.
If I stick with a single stream, even with my custom pipeline, I don't seem to have any problem, it just keeps running.
CPU resources are not an issue - I'm running this on a relative new 6 core Mac.
Brent.
Ullaakut
commented
5 years ago Thanks for all the precisions and even taking the initiative to try it with a different pipeline! Your help is very much appreciated :)
Mh, I don't seem to be able to reproduce the issue. I've tried streaming three videos simultaneously for about 15 minutes, and so far everything seems to work fine. I'll have to try with videos with sound I guess. 🤔
I'll leave it running the whole day and hopefully it will crash at some point. How long does it usually take in your case?
EDIT: After one hour, still running fine EDIT: After five hours. still working 🤔 EDIT: I just stopped them after ~24 hours of continuous streaming, everything worked as expected, so I'll definitely need more information to be able to reproduce this issue.
Ullaakut
commented
5 years ago So since I wasn't able to reproduce the issue, I'm going to need information about:
docker images | grep rtspatt and checking the tag.GST_DEBUG value and pasting the logs here?Thanks 🙏🏽
bpgross
commented
5 years ago I'm sorry for the delay, I've not had much of a chance to debug this - just been dealing with a single stream and re-starting it when it dies.
In terms of version, - I am using latest - says I created it 3 months ago.
For a failure that just happened, I was running a 960x720 video, 15 FPS. It died while I was consuming the video, after about 1450 frames. The video itself is a mp4, 33 MB, 5 minutes long. There was only a single client consuming the video, and I was using port 8558 (-e RTSP_PORT=8558 and -p 8558:8558).
An additional item to be aware of is that I am typically connecting with RTSP over TCP - ie the video comes back over the TCP stream on port 8558 vs a UDP port. I do this because my client consuming the video is in a docker container, and it is just easier to do this than to have to set a range of UDP ports for the docker container that matches what the client side RTSP library will pick.
I did capture a couple of output lines when the streaming failed:
(rtspatt:1): GStreamer-CRITICAL : 06:06:44.559: gst_util_set_object_arg: assertion 'G_IS_OBJECT (object)' failed (rtspatt:1): CRITICAL : 06:06:44.559: gst_app_src_get_max_bytes: assertion 'GST_IS_APP_SRC (appsrc)' failed (rtspatt:1): GLib-GObject-CRITICAL : 06:06:44.559: g_object_set: assertion 'G_IS_OBJECT (object)' failed (rtspatt:1): GLib-GObject-WARNING : 06:06:44.559: invalid (NULL) pointer instance (rtspatt:1): GLib-GObject-CRITICAL : 06:06:44.559: g_signal_connect_data: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed
I had tried running with debug level 3 - but there did not seem to be anything useful in the output. I had tried level 4 but there was way too much output. I can try level 4 tomorrow if that would help and then attach the log file.
Thanks again for all your help so far ...
Brent.
bpgross
commented
5 years ago I spent some time with this over the weekend and I have some further info.
For my use, the resolution of the stream is important as we have videos at specific resolutions to test our analytics. I could not go with the default resolution of 1280x720 25 FPS, and I did not want the CPU load of transcoding to effectively keep the source resolution thus since early on I've been using a custom pipeline as I listed above.
It ends up with a custom pipeline the video does not loop - the message hook to reset the input video at EOS is only added when RTSPATT builds the pipeline. Our videos are of varling lengths thus it was not obvious this was the issue as the streams seemed to "fail" at random points. I was not actually looking at the video output, but rather the output of our analytic engines triggered by video content. The streams will all restart when a new client connects (or the same one reconnects) - they just do not loop with a custom pipeline.
With the above it would be nice if there was a "-norescale" option to just allow the source video to go through without any transcoding, this keeps the CPU usage down on a machine serving up several streams when I want to match the stream resolution to the video resolution.
The other weird thing I could never get working is that when run in a docker container, on my Mac, I could not have RTSP over UDP - I had to use RTSP over TCP (this is something the client requests). With that I could not get multiple docker instances (each running on a different port) to remain up - I did not spend much more time on this - but it may have something to do with the EXPOSE 8554 in the dockerfile - as each running image then wants 8554 and whatever port I use with the -P command line parameter. With RTSP over TCP the video all comes back on the TCP port (eg. 8554) thus there could easily be a conflict if one container is using 8554 for real and another is using 8555 (for example) but has an EXPOSE for 8554.
I ended up changing the cmake file so I could build natively on my Mac, and running that executable directly I was able to use UDP and avoid all the port issues (and UDP performs better) The change to the cmake file was minimal - I used homebrew to install gstreamer and it did not link the includes into /usr/include.
Brent..
Ullaakut
commented
5 years ago Hey @bpgross !
Sorry, I forgot to answer to your last message two weeks ago.
I will change the behavior to not rescale and not change the framerate of the video by default if it's from a file input 👍🏽
This is a bad design from my side since this tool was originally supposed to create videos and not stream existing records.
I'll probably do this either this evening or on Wednesday, and will update this issue once that's done.
As for the EXPOSE instruction within the Dockerfile, I will remove it and that should hopefully help with the second issue.
Thanks again for all of the details on this issue, it's been really helpful 🙏🏽
bpgross
commented
5 years ago Thanks for this fast response. I was going to look into adding a norescale option, but if you are willing to change the default then I do agree that is preferable.
Thanks,
Brent.
Ullaakut
commented
5 years ago @bpgross I've done a few experiments with PR #55 that I opened almost a year ago and was reluctant to merge due to stability concerns, and I think it's ready to enter master now.
Give it a try and let me know if it improves the situation! The issue when new clients consume the stream might still happen, I don't know, I haven't been able to reproduce it at the moment unfortunately
Ullaakut
commented
5 years ago (Make sure to update your image by running docker pull ullaakut/rtspatt:latest)
I would like to stream multiple streams on the same machine - I'm using docker run, and for each one I'm specifying a different input file, different docker container name and different external port (eg. 8554, then 8555, etc.). Docker starts up all my containers successfully, but after a short while all but one stops and I can only ever connect to the first one I start. It looks like there is some sort of conflict with the jack server? I did not want to post detailed output in case this is a known limitation.