Question: there is the possibility to specify a custom ffmpeg set of options to manage the stream?

[dorvan]

There is the possibility to specify a custom ffmpeg set of options to manage the stream? for example we use:

width 2560
height 1440
log_level 9
log_type all
logfile /Data/Data04/motion/motion.log
auto_brightness off
###despeckle_filter EedDl
event_gap 300
ffmpeg_bps 1500000
ffmpeg_duplicate_frames off
ffmpeg_output_movies on
ffmpeg_variable_bitrate 0
ffmpeg_video_codec mp4
framerate 30
input -1
lightswitch 0
locate_motion_mode off
locate_motion_style box
max_movie_time 400
minimum_frame_time 0
minimum_motion_frames 3
netcam_keepalive on
netcam_tolerant_check off
noise_level 32
noise_tune off
norm 0
output_pictures off
quality 100
picture_type jpeg
power_line_frequency -1
pre_capture 10
post_capture 0
process_id_file /var/run/motion/motion.pid
quiet on
setup_mode off
smart_mask_speed 0
threshold 115000
threshold_tune off
webcontrol_html_output on
webcontrol_localhost off
webcontrol_port 7777
### enable extpipes to provide to tosiara a really case ###
use_extpipe on
## extpipe /usr/bin/x264 - --input-res 2560x1440 --fps %fps --bitrate 1500 --preset ultrafast -tune zerolatency --quiet -o %f.mp4
##extpipe /usr/bin/x265 - --input-res 2560x1440 --fps %fps --bitrate 1500 --preset ultrafast -tune zerolatency --quiet -o %f.mp4
### enable extpipes to provide to tosiara a really case ###
extpipe /usr/bin/ffmpeg -re -loglevel warning - -c copy -preset ultrafast -tune zerolatency %f.mp4
camera /cfg/controller/motion/01.conf
camera /cfg/controller/motion/02.conf
camera /cfg/controller/motion/03.conf

The use of extpipes cause a segmentation fault.

[81329.902160] nc2:02[21546]: segfault at 30 ip 00007f48e9ec816b sp 00007f48d37fda80 error 4 in libavformat.so.57.56.101[7f48e9d7c000+1ca000]
[81602.910728] nc2:02[22106]: segfault at 20 ip 00007f25a6815510 sp 00007f25857f20f8 error 4 in libavformat.so.57.56.101[7f25a67c1000+1ca000]
[81657.916363] nc2:02[22135]: segfault at 30 ip 00007fefa605716b sp 00007fef87010a80 error 4 in libavformat.so.57.56.101[7fefa5f0b000+1ca000]
[82202.891960] nc2:02[22968]: segfault at 1df2 ip 00007f1ae182c239 sp 00007f1a9d7f97c0 error 6 in libavcodec.so.57.64.101[7f1ae15fa000+c29000]

camera conf:

netcam_url rtsp://test:test@10.2.22.2:554/h264Preview_01_main
##netcam_url rtmp://10.2.22.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=test&password=test
### rtsp syntax test
### netcam_url rtsp://10.2.22.2:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=test&password=test
target_dir /Data/Data02/RecordingStorage/02/
movie_filename rec-%v-%Y%m%d%H%M%S
picture_filename rec-%v-%Y%m%d%H%M%S
snapshot_filename rec-%v-%Y%m%d%H%M%S
timelapse_filename rec-%v-%Y%m%d-timelapse

as default, yuv420p is used to process, we need to change it.

[2:ml2:02] [INF] [ENC] [Jun 19 10:09:54] ffmpeg_avcodec_log: setting jitter buffer size to 0
[2:ml2:02] [INF] [ENC] [Jun 19 10:09:54] ffmpeg_avcodec_log: Reinit context to 2560x1440, pix_fmt: yuv420p

[tosiara]

How can extpipe cause a segfault if it is disabled in your config? use_extpipe off

[dorvan]

using it, this is a running config in with extpipes disabled.... if you want i can fix it to show how appears when are used (removing comments and make extpipes on)

[tosiara]

Yes please, provide a real runtime config

[dorvan]

fixed, reload page please

[dorvan]

seg fault it's not related to extpipe command specified, when extpipes are used libavcodec seg fault appear always. there is something wrong in the conf?

[tosiara]

So when does segafault happen? Immediately after you started motion? When event is triggered? Can you provide full motion log from start till segfault?

[dorvan]

after an event is triggered.

[dorvan]

i can provide, but firstly we can discuss on a irc or similar? due provide a coeherent test also with correct params?

[tosiara]

Sorry, I cannot help you on irc right now.

[dorvan]

ok, but we obtain a fragmented (gray, green stripes, frozen sections) and delayed not smooth videos and downscaled respect about h/w defined considering the bitrate it's around available 1900kB to 8192k we want to obtain a native 4MP video in mp4 format with a pre record of event of 10s to all event for a maximum of 400s video frames are more 30fps. This params are the common default to put into motion.conf (so i think we need to review/correct), the on a working and stable conf, we want make 3 tests, divide bitrate for camera (2 cam are on 8192kB/s and the third on 2048kB/s), use external pipes to provide custom settings to ffmpeg on rtsp, then switch rtmp (already answered question by you, thanks) with this, sure about global conf i can provide you all logs on 3 tests to check seg fault

[tosiara]

If possible, please provide full motion log from beginning

[tosiara]

By the way, I have tried your extpipe command and it does not work for me:

./ffmpeg -re -loglevel warning - -c copy -preset ultrafast -tune zerolatency %f.mp4
Option re (read input at native frame rate) cannot be applied to output url - -- you are trying to apply an input option to an output file or vice versa. Move this option before the file it belongs to.
Error parsing options for output file -.
Error opening output files: Invalid argument

[dorvan]

@tosiara because it's wrong! as extpipe(as reported in examples), "-" inject inputs and params passed by motion (this have absolute sense) and being expanded on execution. As complete standalone command you have to substitute "-" with motion provided params

so:

ffmpeg -re -loglevel warning -i "rtsp://test:test@10.2.22.2:554/h264Preview_01_main" -c copy -preset ultrafast -tune zerolatency -t 05:00 -y /Data/Data02/RecordingStorage/02/rec-02-example.mp4

let's try.

in motion extpipe "-" seems pass the params to become: "-i rtsp://test:test@10.2.22.2:554/h264Preview_01_main"

[tosiara]

No, you don't need to substitute anything. "-" specifies to read stdin. Motion writes raw frames to stdout so your extpipe command read frames and encodes them. What you want is to read rtsp stream and record it. You should do this not using extpipe, but using event

[dorvan]

I want to use a custom command line to launch ffmpeg, instead of what it's cabled in motion... all "Per-event trigger" seems to make a thread forking looking to an event, are not customization of ffmpeg parameters. I've underlined what you have trying to run on ffmpeg, as I have understood here: motion.conf i have to put " - " to say to motion "where expand motion.conf per-cameras passed data" like a variable or a regex. Is it wrong?

how I can make it?

Please make me an example of globally defined extpipe called and used by motion, completing extpipe with per-camera settings for each thread.

Thanks. solved this ( I can have a good log to pass you entirely )

[tosiara]

You have to declare your command like on_event_start ffmpeg blah blah blah

[dorvan]

but on_event_start is it a conditional OR or an AND respect the the cabled-in trigger for ffmpeg thread in motion?

seems looking to motion process thread forking:

on_event_start == AND extpipe == OR

example:

on_event_start: I can execute a "mail send" to advise me of an motion event, but motion record always the movie of the event (AND), in case of ffmpeg i have 2 ffmpeg triggers to make the movie not one.

extpipes: I can excute and external program instead of the ffmpeg built in trigger, i can also use ffmpeg with different "core" options, passing camera parameters.

Can you make me and example on 2 option differences on thread/trigger forking? Have I understood it in wrong way? Can you explain the behavior of both options, more than specified here ?

[tosiara]

I didn't get your OR/AND part

Both scripts will execute when motion is detected. But extpipe is designed to receive DECOMPRESSED stream into stdin, it should not be used to send emails or any other external commands. Just turn it off if you are not going to record UNCOMPRESSED stream (stream that motion already captured and decoded, with timestamps and motion box). If you want to send email or run ffmpeg to record raw compressed stream directly from your camera - you should put your command or script name into on_event_start

More to say, based on your provided config you are also using internal encoder to save videos: ffmpeg_output_movies on. So you are now trying to record the stream 3 (three) times. Please check once again what do you want to achieve with motion. At the moment it is a bit mess

[dorvan]

I didn't get your OR/AND part

This is a problem if I have 2 different options probably make 2 different things:

reading the documentation seems looks like:

What is motion:

Motion it's a program who stay looking for frame differences on a source "aka event" and trigger something. works on ffmpeg/libavcodec. and some triggers called by conf options works as OR (on event, make this OR this) and other trigger works as AND (on event, make this AND this)

If I do not specify triggers for external commands (on_* options, and extpipes) only internally cabled triggers are used.

So:

What's the default behaviour of motion?

On event (aka motion event, found by frame diffs check) trigger internally the making of a movie using ffmpeg directly, and pictures (ex: to provide thumbnails related to the event).

Optionally I can make motion conf able to send me an advise (sms, mail, text, banner, poke) triggering an external application using on_* configuration options

EXAMPLE about AND/OR dilemma: with on_event_start i can start: echo "hey, I catched up something changed" && date > /Data/events.log I obtain: 3 media about the event: 1 movie, 1 picture AND 1 /Data/events.log new line is this correct?

looking to extpipes: I can specify an "alternate command, instead of internally called ffmpeg avcodec to make the movie" (1 of default outputs of motion). This SUBSTITUTE the default call to make movie output. so this, technically it's an OR Use default command line OR customized one. is this correct?

[tosiara]

What do you want to happen when motion detection occurs?

[dorvan]

Event dependency/trigger, and movie capture string build: motion -> event detection -> motion internally triggered ffmpeg + my custom params + motion conf defined params + camera input path from secondary "camera threads" defined in motion.conf sub files-> output media = 1 movie (*with no additional `on_` overhead or externally triggered ffmpeg or other** ) (extpipe do this?).

if motion internally triggers ffmpeg passing params, building command syntax, the need it's a convertion specifiers to inject custom params in the movie command string.

Example: Injected custom params: "{ }" motion.conf derived value expansion: "Bold" motion.conf already present value expansion with ConversionSpecifiers: "Italic"

ffmpeg {-re -loglevel warning} -i "rtsp://test:test@10.2.22.2:554/h264Preview_01_main" {-c copy -preset ultrafast -tune zerolatency} -t 05:00 /Data/Data02/RecordingStorage/02/rec- %v-%Y%m%d%H%M%S

Please, confirm or deny my is this correct? questions, to correct me if the reality it's different.

[dorvan]

@tosiara are you there?

[tosiara]

motion internally triggered ffmpeg + my custom params + motion conf defined params + camera input path from secondary "camera threads" defined in motion.conf sub files-> output media

Sorry, but I don't understand what you want to achieve After motion event happened you have following options that ALL can be executed at the same time (is that what you called "AND"?):

0. save picture (output_pictures on)
1. encode stream using motion's internal encoder (ffmpeg_output_movies on)
2. pipe uncompressed stream to external encoder (use_extpipe on)
3. run script and do whatever you want using event command (on_event_start my_script.sh)

You can have any combination of 0,1,2,3. OR, AND, or none

[dorvan]

Finally:

extpipe: as you wrote (now), and as I wrote yesterday, it's the correct way. Because:

@tosiara cit:

1. encode stream using motion's internal encoder
2. pipe uncompressed stream to external encoder (extpipe) 

so:

2. it's the correct way, extpipe because it make a pipe to external command for motion in-proc movie thread.

On other hand:

@tosiara cit

3. run script and do whatever you want using event command 

It's right! So it's does not make a pipe to, but triggers a fork leaving working for itself the in-proc movie thread. So motion it's called to make 2 things not one (overhead). I don't want it.

BUT how the extpipe it's filled by camera parameters (coming from motion.conf) about stream url, user, password? extpipe command (as STDIN flux) specification it's used as command prefix, suffix or like the example reported here

Where, about extpipe:

 # External program (full path and opts) to pipe raw video to
 # Generally, use '-' for STDIN...

so where ' - ' it's used like a ConversionSpecifiers to inject motion.conf camera input params?

Why ' - ' it's not present or documented here ?

How, in the extpipe command being build/expanded/injected the input parameters present in motion.conf and cameras?

it's clear in the related documentation this it's not documented.

Please, make an example.

[tosiara]

Extpipe does not pass any parameters - it only pipes uncompressed frames that are already captured by motion (netcam_url). You can only specify params for encoder what to do with this stream. Ex, what codec use. The extpipe command DOES NOT connect to camera, and you DON'T NEED TO SPECIFY USER OR PASSWORD. Uncompressed frames are provided by motion

[tosiara]

There are examples in the default config:

https://github.com/Motion-Project/motion/blob/master/motion-dist.conf.in#L341

[dorvan]

Extpipe does not pass any parameters.

OK, i know, but i think motion, inject ffmpeg -i params from motion.conf to extpipe specification.

it only pipes uncompressed frames that are already captured by motion

OK!!!

So motion does NOT complete the extpipe command to launch it.... so it's not a "command - config pipe" but really a "movie stream pipe"

ok so at today i cannot specify different custom options to motion to encoding the movie output with the default ffmpeg to build the custom command.

I can only re-encode the movie, playing on stream pipe. Right?

[dorvan]

in the extpipe example:

the specified input have to follow the ID of the camera thread? -i pipe:0 == camera thread so (%v) ? or it's always 0 ???

[tosiara]

Internal encoder accepts only following params: ffmpeg_video_codec XXX, ffmpeg_variable_bitrate YYY, ffmpeg_bps ZZZ

[tosiara]

You don't need to change -i pipe:0, it is encoder's specific command line If you want to understand how encoder reads the stream from stdin, check this command, it is simpler: https://github.com/Motion-Project/motion/blob/master/motion-dist.conf.in#L339

[dorvan]

already tried but it's not reliable.

[tosiara]

what is not reliable?

[dorvan]

thread forking and triggered external ffmpeg process management..

[tosiara]

thread forking and triggered external ffmpeg process management

what????

[dorvan]

it's slow if i set tup use extpipes, really slow.

[tosiara]

Extpipe is the fastest available option to encode movies. If it is slow then probably you have to consider upgrading your server's hardware

[dorvan]

@tosiara it's true. It's not a motion related problem, it's related to server load, and related to streams quality.

there are 6 streams (4MP), 3 managed by motion, 3 started directly from ffmpeg for "live" streams, 2 our applications, +1 VM on a small 4core x86 mini-pc ...

BUT on a events on 3 cameras thread forking cause 9 threads on extpipes. Cause a slow start on motion capture. In our env.

We are here to ask something to take the best from motion, independently the hw platform. Already asked questions was never defensive or offensive or something different to explain reality and take help about questions.

on your side seems appear an always defensive position, instead of helpfull position.

... Like the on_* event discussion where i've asked something totally different. ... Like the rstp..

or about my asks to confirm or deny if I've correctly understood explaining my answers to your sentences.

[iskunk]

@dorvan: 4MP streams are quite heavyweight, especially if you want a reasonable framerate. I would suggest setting the cameras to something modest like 640x480, and see how well that works. If that works fine, but the same config at 4MP chokes, then the hardware is just not up to snuff.

Alternately, you may want to have a look at #124, where a new "passthru recording" feature is being developed. This will allow recording streams exactly as they come in, with no re-encoding overhead. One of the motivations for this work, in fact, is to make it possible to record hi-res streams without needing such beefy hardware.

[dorvan]

@iskunk thanks for answer..

I can add to "motivations for this work, in fact" it's to use motion as "motion detection and streams gateway" to re-encoding as-needed with external programs independently from supported protocols, rstp on some net camera firmwares it's buggy, rtmp make a more fluent e cpu-friendly protocol, more friendly with multiple streams encoding (with same source) for example to provide recordings and together live view, without beefy hardware.

[dorvan]

@iskunk yes 4MP streams are not easy to manage, but looking to network interface when we speak about 4MP (3.7 real, without control bits) we considering a (already coded/compressed, usually by a ffmpeg and served by an nginx) 8-12 Mbyte/s stream, usually on a up-to a 4MP netcam the hardware asset it's always defined around a 100Mbit/s (w/o full duplex) network interface, up to 200Mbit/s bandwidth on WiFi Cameras. So this the poorest IP hardware today and the mainstream of all network surveillance products. Top Class products (>6MP) works on a 300Mbit/s for Wifi (1x 20Mhz channel slot) and 500/700 Mbit/s for cabled considering 1 Gigabit/s interface. so a resolution or coding depending gateway it's not a practicable way to support all streams. The stream manipulation have to be resolution independent as possible and more related to bandwidth, or I/O related (stream management concurrency, recording concurrency) because the slowest component on the system is the real storage access bandwidth.

[iskunk]

Network bandwidth is usually not the limiting factor when everything is on the same LAN. The problem is CPU usage. (1) Decoding, (2) encoding, and (3) motion detection all eat up CPU, and the usage tends to be directly proportional to the image resolution.

Passthru recording gets rid of (2) entirely. There has been discussion of implementing motion-vector-based detection in Motion, which would reduce (3) and possibly (1) significantly, but is still just a pipe dream.

[dorvan]

@iskunk so you have answered to @tosiara about why I have opened this issue. THANKS.

"If you can not be heard, you just have to make sure that someone, who is listened, comes to your own conclusion"

[dorvan]

workaround (or porkaround)

motion.conf

daemon off
width 2560
height 1440
log_level 1
log_type all
framerate 30
ffmpeg_output_movies off
ffmpeg_video_codec mp4
input -1
netcam_keepalive on
netcam_tolerant_check on
picture_type jpeg
output_pictures off
pre_capture 10
event_gap 310
max_movie_time 310
post_capture 0
threshold 110000
webcontrol_html_output on
webcontrol_localhost on
webcontrol_port 7777
logfile /Data/Data04/motion/motion.log
process_id_file /var/run/motion/motion.pid
camera /cfg/anthilla/motion/01.conf
camera /cfg/anthilla/motion/02.conf
camera /cfg/anthilla/motion/03.conf
camera_dir /cfg/anthilla/motion/conf.d

a camera conf file

ffmpeg_bps 8192000
camera_name 01
netcam_url rtsp://test:test@10.2.22.1:554/h264Preview_01_main
target_dir /Data/Data02/RecordingStorage/01/
movie_filename rec-01-%Y%m%d%H%M%S
picture_filename rec-01-%Y%m%d%H%M%S
snapshot_filename rec-01-%Y%m%d%H%M%S
timelapse_filename rec-01-%Y%m%d-timelapse
use_extpipe on
extpipe /usr/bin/ffmpeg -re -hide_banner -loglevel panic -y -i 'rtmp://10.2.22.1:1935/bcs/channel0_main.bcs?channel=0&stream=0&user=test&password=test' -c copy -preset ultrafast -tune zerolatency %f.mp4

@iskunk you understand why I gnore the pipe stream? directly with ffmpeg options I remove a stream encoding/decoding of computed motion stream, recording directly the native coded and compressed stream format. in this way i save 30% of cpu allocation, and 40% of I/O

video are recorded smoothly on a cheap hardware with possibility to manage 8 streams (4 rec and 4 live+ webcam ftp transfer + applicative server +1 vm) easily.

[tosiara]

I still recommend you using event script instead of extpipe - you will save few more cpu

[dorvan]

@tosiara thanks, like i have asked before, can you make me an example?

[tosiara]

I don't have an example as I don't use scripts. Try using the same command, but add it to on_event_start instead

[dorvan]

We (as I already wrote before) initially used tested all on_*_start params, yes, CPU usage it's lower, but video recording are unpredictive, and have a slow start... extpipesseems a good combination, but I can retry and verify more deeply.

...without a script, we prefer to not call command shell or GNU core utils to trigger an event... console and coreutils adds a delay and resource usage.

we have tryied with,

• console scripts,
• binary triggers,
• calling init system

[tosiara]

You want to say that extpipe command starts faster than on_event_start command even though the command is identical? If this is the case, can you provide more details how you measure the lag and steps to reproduce. Technically both them spawns the same process and there should not be difference in timing

[dorvan]

strange:

tosiara commented 8 days ago Extpipe is the fastest available option to encode movies. If it is slow then probably you have to consider upgrading your server's hardware

this is true. I can think how to provide evidences, but why this your answer? @tosiara check your position.

I've wrote:

dorvan commented 8 days ago it's slow if i set tup use extpipes, really slow.

becase i use the extpipe, to re-encode the pipe source, now i ignore it, use extpipe as trigger and call the rtmp directly at 4MP, mp4, without re-encoding.

This is more direct and managed instead other on-*-start triggers

[tosiara]

You don't have time stamp and motion location box in your directly recorded videos. If you want them - you have to encode. I was talking about the fastest way to encode. extpipe is the fastest and most flexible way. "Fast" means better cpu performance of frames per second.

You say that you have "slow start" of on_event_start. I asked you to provide an evidence of "slow start"

Slow start and fast encoding are completely different things. Which one you wanna talk about?