Motion's sub stream

[tosiara]

Anyone interested in having secondary live stream with different resolution and rate? And/or maybe introduce resize option of the live stream?

[iskunk]

This would be an MJPEG sub stream generated by Motion itself, not one coming from the camera?

[dorvan]

no.... why a MJPEG and not a direct stream?

[tosiara]

@iskunk it will the same mjpeg stream as we already have implemented, but just with different resolution and/or framerate. Substream with pass-through would be also nice, but doesn't look trivial so far. And also, the live stream would really need motion location box, text output and other stuff that we lack in pas through

[dorvan]

@tosiara I suggest to think on a substream that permits a "realtime" live views usage as source for with the HLS (Http Live Straming)

[tosiara]

For realtime live stream I can view directly from camera, no need to pipe it through the motion. The primary reason for adding sub stream within motion is:

1. have camera text and time stamp on the frame
2. have motion location box
3. reduce resolution and bandwidth to watch on mobile device

If you don't need them - you can watch live stream straight from the camera

[dorvan]

A substream tu use as live stream gateway from camera it's usefull to put on a web application like web NVR the live view of the network camera adding features like motion alert event. Realtime it's the same of "get and forward captured data asap, not make more processing on the stream, period"

[Mr-Dave]

I am unclear on why this would be needed. In particular, why would we need TWO streams. The indicated items so far would be seem to be more of enhancements to the existing stream.

[tosiara]

The main idea was to have multiple live streams on the same html page (http://lavrsen.dk/foswiki/bin/view/Motion/FrequentlyAskedQuestions#How_do_I_see_more_than_one_camera_stream_at_a_time_63). For me 8 streams 480p are fine, you only need to tune browser setting to increase persistent connections if cameras are on the same server. But for 720p streams it generates significant traffic and becomes almost impossible to view on mobile over 3g. Also, 720p MJPEG encoding on the motion side significantly increases CPU (serious issue for low power ARM). So the idea was to create a secondary low res stream on another port to save bandwidth, save CPU and still be able to view all the cameras at the same time

[iskunk]

If mobile / low-bandwidth scenarios are the goal, something I've thought would be nice is to have Motion generate a montage view in a single stream, and show that on the Web UI overview page. Use HTML image-maps to make the individual camera views clickable as they are now.

Not only would that cut down the bandwidth, it also renders moot the number-of-connections issue that folks often run into.

(Oh, and welcome back, @Mr-Dave :-) Please look at some of my open issues! In particular, the passthru-recording work is currently stalled on the motion-box-subtitle-overlay stuff, but there's already plenty that can start making its way into master...)

[Mr-Dave]

I'm still not quite understanding. The current webcontrol page allows users to see all the images on a single page and scale them as they wish which I believe addresses the underlying issue in the linked FAQ (This is also duplicated in our FAQ)

The other issues seem contradictory. My guess is that rescaling the image and then putting that out as a additional stream is going to increase CPU usage not decrease it.

[jogu]

The current webcontrol page allows users to see all the images on a single page and scale them as they wish

This scaling is done in the browser - it doesn't reduce the amount of network traffic. Network traffic is an issue when viewing cameras (particularly high frame rate or high resolution ones) remotely, particularly over a 3G connection.

The other issues seem contradictory. My guess is that rescaling the image and then putting that out as a additional stream is going to increase CPU usage not decrease it.

It will increase CPU usage in some scenarios and decrease it in others.

I'd personally find this feature useful, as I do view my cameras over slow mobile connections and it often doesn't work or takes a long time at the moment.

For a decrease CPU scenario:

If the user is viewing the current index page, with the images scaled down by the browser, we're doing jpeg compression on the full stream. If we scale the images down first, we're only doing jpeg compression on smaller images, which will be faster. (Assuming all other things are equal; in multiple user scenarios you may have users viewing both sizes at the same time which increases CPU usage. Also may vary depending on the jpeg quality setting.)

[tosiara]

Yes, @jogu is right. When viewing through the webcontrol, the browser receives the full image and scales it to 25% at the client side. This isn't reducing the bandwidth Regarding CPU usage, from what I see, connecting to the stream served by low powerARM device, MJPEG encoding adds 15-25% of cpu to the main thread. If the same thread is already close to 60-70% with decompression and motion detection you will start dropping frames. Resizing uncompressed 720p to 480p will add just 1-2%, right?

I don't have exact numbers right now. But as I have a working prototype in my repo I will share it and perform all the needed PoC benchmarks so we can discuss further. Unless anyone else volunteers to implement it

[tosiara]

While looking at the code it looks like we encode every frame to jpeg and only put them to the socket at the specified rate. Which is not quite efficient, I believe

I applied this patch to increase logging:

diff --git a/event.c b/event.c
index 5c4bdfc..e68a112 100644
--- a/event.c
+++ b/event.c
@@ -343,7 +343,10 @@ static void event_stream_put(struct context *cnt,
             void *dummy2 ATTRIBUTE_UNUSED, struct timeval *tv1 ATTRIBUTE_UNUSED)
 {
     if (cnt->conf.stream_port)
+    {
+        MOTION_LOG(NTC, TYPE_EVENTS, NO_ERRNO, "Stream put");
         stream_put(cnt, img);
+    }
 }

diff --git a/motion.c b/motion.c
index 99c9500..a43da45 100644
--- a/motion.c
+++ b/motion.c
@@ -2325,8 +2325,11 @@ static void mlp_loopback(struct context *cnt){
               &cnt->pipe, &cnt->current_image->timestamp_tv);

         if (!cnt->conf.stream_motion || cnt->shots == 1)
+        {
+            MOTION_LOG(NTC, TYPE_ALL, NO_ERRNO, "EVENT_STREAM: %d %d", cnt->shots, cnt->conf.stream_motion);
             event(cnt, EVENT_STREAM, cnt->current_image->image, NULL, NULL,
                   &cnt->current_image->timestamp_tv);
+        }
     }

     event(cnt, EVENT_IMAGEM, cnt->imgs.out, NULL, &cnt->mpipe, &cnt->current_image->timestamp_tv);
diff --git a/stream.c b/stream.c
index 3de3fb2..73476e2 100644
--- a/stream.c
+++ b/stream.c
@@ -847,6 +847,7 @@ static void stream_flush(struct stream *list, int *stream_count, int lim)
                  * 'filepos' contains how much of the buffer
                  * has already been written.
                  */
+                MOTION_LOG(NTC, TYPE_STREAM, NO_ERRNO, "Write");
                 written = write(client->socket,
                           client->tmpbuffer->ptr + client->filepos,
                           client->tmpbuffer->size - client->filepos);
@@ -1222,6 +1223,9 @@ void stream_put(struct context *cnt, unsigned char *image)
             /* Update our working pointer to point past header. */
             wptr += headlength;

+            /* WIP Resize image if request was not "/" */
+            MOTION_LOG(NTC, TYPE_STREAM, NO_ERRNO, "Picture put: %d", cnt->imgs.size);
+
             /* Create a jpeg image and place into tmpbuffer. */
             tmpbuffer->size = put_picture_memory(cnt, wptr, cnt->imgs.size, image,
                                                  cnt->conf.stream_quality);

And here what I get: https://gist.github.com/tosiara/d60afe4077214f4826480fb38cce4919

Before the client connects to the stream you should see calls to event_stream_put for every frame:

[1:ml1] [NTC] [ALL] mlp_loopback: EVENT_STREAM: 3 0
[1:ml1] [NTC] [EVT] event_stream_put: Stream put

Then when client connects there is a sequence:

[1:ml1] [NTC] [STR] stream_put: Picture put: 1382400
[1:ml1] [NTC] [STR] stream_flush: Write
[1:ml1] [NTC] [ALL] mlp_loopback: EVENT_STREAM: 1 0
[1:ml1] [NTC] [EVT] event_stream_put: Stream put
[1:ml1] [NTC] [STR] stream_put: Picture put: 1382400
[1:ml1] [NTC] [ALL] mlp_loopback: EVENT_STREAM: 2 0
[1:ml1] [NTC] [EVT] event_stream_put: Stream put
[1:ml1] [NTC] [STR] stream_put: Picture put: 1382400
[1:ml1] [NTC] [ALL] mlp_loopback: EVENT_STREAM: 3 0
[1:ml1] [NTC] [EVT] event_stream_put: Stream put
[1:ml1] [NTC] [STR] stream_put: Picture put: 1382400
[1:ml1] [NTC] [ALL] mlp_loopback: EVENT_STREAM: 4 0
[1:ml1] [NTC] [EVT] event_stream_put: Stream put
[1:ml1] [NTC] [STR] stream_put: Picture put: 1382400
[1:ml1] [NTC] [ALL] mlp_loopback: EVENT_STREAM: 5 0
[1:ml1] [NTC] [EVT] event_stream_put: Stream put
[1:ml1] [NTC] [STR] stream_put: Picture put: 1382400
[1:ml1] [NTC] [ALL] mlp_loopback: EVENT_STREAM: 6 0
[1:ml1] [NTC] [EVT] event_stream_put: Stream put
[1:ml1] [NTC] [STR] stream_put: Picture put: 1382400
[1:ml1] [NTC] [ALL] mlp_loopback: EVENT_STREAM: 7 0
[1:ml1] [NTC] [EVT] event_stream_put: Stream put
[1:ml1] [NTC] [STR] stream_put: Picture put: 1382400
[1:ml1] [NTC] [ALL] mlp_loopback: EVENT_STREAM: 8 0
[1:ml1] [NTC] [EVT] event_stream_put: Stream put
[1:ml1] [NTC] [STR] stream_put: Picture put: 1382400
[1:ml1] [NTC] [ALL] mlp_loopback: EVENT_STREAM: 0 0
[1:ml1] [NTC] [EVT] event_stream_put: Stream put
[1:ml1] [NTC] [STR] stream_put: Picture put: 1382400
[1:ml1] [NTC] [ALL] mlp_loopback: EVENT_STREAM: 1 0
[1:ml1] [NTC] [EVT] event_stream_put: Stream put
[1:ml1] [NTC] [STR] stream_put: Picture put: 1382400
[1:ml1] [NTC] [STR] stream_flush: Write

My camrea runs at 10fps, so clearly we write to socket at 1fps which is fine, but we call put_picture_memory for every frame at 10fps which is very inefficient

[tosiara]

Ah, I see now. Motion assumes that there are multiple clients reading the stream and connected at different time. So every client needs to get his frame accordingly to frame rate. So I understand there is nothing we can do, there always must be a frame encoded and ready to send to a client. Oh well... Unless we redesign everything using Libmicrohttpd there is a lot of work that needs to be done to optimize the process

[jogu]

Yes.

I would suggest the easy change is to allow a second port (and hence thread) for each camera that runs at a scale/size specified in the config file.

[tosiara]

I'm still puzzled, so why cpu spikes as soon as a client connects to the stream? If a frame is anyway encoded what is using cpu?

[tosiara]

Ahh, sorry. Picture is only encoded when at least one client is connected

[tosiara]

Making progress a little bit. Now have separate socket for the secondary stream. Getting closer to resize. My test branch: https://github.com/tosiara/motion/tree/wip-secondary-stream2

[tosiara]

A prototype is working and ready to test. Substream listens on a separate port and provides a picture scaled down by 50% and at the same rate which is configured for the main stream Clone my working tree: https://github.com/tosiara/motion/tree/wip-secondary-stream2 Put this config option into your config: substream_port 8082

Quick benchmarking 1280x720 V4L MJPEG @10 fps and default 1fps stream gave me the following results.

• Without substream (commented out in config) Idle motion detection: ~57% CPU One client viewing full stream: ~82% CPU Bandwidth: 24 KB/s

• With substream (uncommented option substream_port 8082) Idle motion detection: ~60% CPU (+3% CPU) One client viewing substream: ~68% CPU (-14% CPU) Bandwidth: 7.4 KB/s (-16.6 KB or -70%)

Still can be optimized to avoid of processing every frame. Also stream code can be further rewritten to make some functions context independent.

Open for your comments and suggestions

[tosiara]

And I have just pushed one more change that will resize or encode stream's frames only if there are clients connected. This saves additional CPU that introduced by substream (no more +3% CPU overhead), only benefits! :)

[Mr-DaveDev]

I am still not clear on why we need TWO streams rather than putting the scaling / configuration options on the existing stream we have.

[jogu]

The current architecture of the streams is to send same data to all connected clients. Or am I missing a way to add a scale as a parameter the client supplies without major rework?

Or maybe I misunderstood you question and the answer is 'because people want to still have the choice of viewing the full res stream'?

[Mr-DaveDev]

It is the second question and I'm trying to determine what would be the the plan for when the users come back and indicate they want/need a third stream for their Wifi computer, and a fourth substream for their work computer and a fifth substream for when they are at the friends house and a sixth substream that provides the images in a different format.

[jogu]

I think to go further than a stream & a substream we'd need a more flexible architecture using a proper http server.

This way should cover 90%+ of the use cases. Most IP cameras also provide exactly two streams - a full one and a low bandwidth one.

[tosiara]

@Mr-Dave things are a bit complicated with the current stream implementation that require a lot of efforts to make the stream flexible so you would not require a secondary stream and just control the existing one.

I started from trying to make the stream depend on user's request (https://github.com/tosiara/motion/commit/39924667d5b8160d2c1130d7d9c70191c9df38f4). So you can dynamically ask, ex, http://localhost:8081/?resolution=320x240 and get lower resolution on the existing stream. However, it turned out that motion continuously writes every frame to the stream's buffer and then sends out current frame to any connected client based on timer. In this case we would need to allocate a buffer for every new client and resize every frame based on client's request.

Another thing is that motion does not expect a delay while parsing client's request. In this case if client is slow (like on mobile) motion throws 'bad request' and disconnects. This is inconvenient and may break existing user's expectation. So I gave up the idea of flexible streams - too much of architectural changes

However, I did implement a secondary stream with a half scaled frame as a proof of concept and I'm pretty happy now running it on my system. I watch 8 streams on a single HTML page like this:

<a href="http://camera01:8081"><img src="http://camera01:8082"></a>

This way you see small preview and after clicking on the preview you can see a single full res stream if needed This way I noticeably save network bandwidth on my 3G connection and also save some CPU on the motion device which now saves me FPS. Previously, when live viewing 720p the recording FPS was dropping because of CPU close to 100%. The results are really cool and I have absolutely resolved all my problems

AFAIK, there no real plans yet to move to libmicrohttpd which would allow to rewrite the stream's code and make it flexible. So for now I will continue using my "temporary hack" and users are welcome to use it too and suggest improvements.

TL;DR; This PR https://github.com/Motion-Project/motion/pull/454 has been created for visibility and open for comments/suggestions. I'm not forcing anyone to merge this as is. If the community decides this change is rather harmful than useful - we will close this PR