MJPEG incoming source encoding

[immek]

Hello, thank you for your great work on this project!

I am using 1.8.2 with Frigate and currently I'm struggling with incoming mjpeg stream.

This configuration works:

entrance:
    - exec:ffmpeg ...
        -filter_complex 
            "[0:v]...split=2[v1][v2]; 
                [v1]...[v1s]; 
                [v2]...[v2s]" 
            -map "[v1s]" ...
              -rtsp_transport tcp -f rtsp rtsp://172.17.0.1:8554/entrance-full 
            -map "[v2s]" ...
              -rtsp_transport tcp -f rtsp {{output}} 
entrance-full:
entrance-mjpeg:
    - ffmpeg:entrance#video=mjpeg#raw=-q:v 15 -tune:v zerolatency -tune fastdecode

The running mjpeg process:

ffmpeg -hide_banner -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://127.0.0.1:8554/entrance?video -q:v 15 -tune:v zerolatency -tune fastdecode -c:v mjpeg -an -f mjpeg -

ffprobe shows:

[NULL @ 0x55d9b0734140] Opening 'http://imms10:5000/live/webrtc/api/stream.mjpeg?src=entrance-mjpeg' for reading
...
[http @ 0x55d9b0734a80] request: GET /live/webrtc/api/stream.mjpeg?src=entrance-mjpeg HTTP/1.1
User-Agent: Lavf/59.27.100
Accept: */*
Range: bytes=0-
Connection: close
Host: imms10:5000
Icy-MetaData: 1

[mpjpeg @ 0x55d9b0734140] Format mpjpeg probed with size=2048 and score=100
[mpjpeg @ 0x55d9b0734140] Before avformat_find_stream_info() pos: 0 bytes read:3747 seeks:0 nb_streams:1
[mjpeg @ 0x55d9b0738a00] marker=d8 avail_size_in_buf=7250
[mjpeg @ 0x55d9b0738a00] marker parser used 0 bytes (0 bits)
[mjpeg @ 0x55d9b0738a00] marker=fe avail_size_in_buf=7248
[mjpeg @ 0x55d9b0738a00] marker parser used 16 bytes (128 bits)
[mjpeg @ 0x55d9b0738a00] marker=db avail_size_in_buf=7230
[mjpeg @ 0x55d9b0738a00] index=0
[mjpeg @ 0x55d9b0738a00] qscale[0]: 20
[mjpeg @ 0x55d9b0738a00] marker parser used 67 bytes (536 bits)
[mjpeg @ 0x55d9b0738a00] marker=c4 avail_size_in_buf=7161
[mjpeg @ 0x55d9b0738a00] marker parser used 0 bytes (0 bits)
[mjpeg @ 0x55d9b0738a00] marker=c0 avail_size_in_buf=7048
[mjpeg @ 0x55d9b0738a00] Changing bps from 0 to 8
[mjpeg @ 0x55d9b0738a00] sof0: picture: 640x480
[mjpeg @ 0x55d9b0738a00] component 0 2:2 id: 0 quant:0
[mjpeg @ 0x55d9b0738a00] component 1 1:1 id: 1 quant:0
[mjpeg @ 0x55d9b0738a00] component 2 1:1 id: 2 quant:0
[mjpeg @ 0x55d9b0738a00] pix fmt id 22111100
[mjpeg @ 0x55d9b0738a00] Format yuvj420p chosen by get_format().
[mjpeg @ 0x55d9b0738a00] marker parser used 17 bytes (136 bits)
[mjpeg @ 0x55d9b0738a00] escaping removed 26 bytes
[mjpeg @ 0x55d9b0738a00] marker=da avail_size_in_buf=7029
[mjpeg @ 0x55d9b0738a00] marker parser used 7003 bytes (56024 bits)
[mjpeg @ 0x55d9b0738a00] marker=d9 avail_size_in_buf=0
[mjpeg @ 0x55d9b0738a00] decode frame unused 0 bytes
[mpjpeg @ 0x55d9b0734140] All info found
[mpjpeg @ 0x55d9b0734140] After avformat_find_stream_info() pos: 7311 bytes read:11931 seeks:0 frames:1
Input #0, mpjpeg, from 'http://imms10:5000/live/webrtc/api/stream.mjpeg?src=entrance-mjpeg':
  Duration: N/A, bitrate: N/A
  Stream #0:0, 1, 1/25: Video: mjpeg (Baseline), 1 reference frame, yuvj420p(pc, bt470bg/unknown/unknown, center), 640x480, 0/1, 25 tbr, 25 tbn
Processing read interval id:0 start:N/A end:N/A
...

But I would like to move the stream definition from entrance-mjpeg to the split in entrance filter and map (split=2 -> split=3). The first step to do so is by leaving configuration entrance-mjpeg empty and trying to run separated ffmpeg process like this:

ffmpeg -hide_banner -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp://172.17.0.1:8554/entrance?video -q:v 15 -tune:v zerolatency -tune fastdecode -c:v mjpeg -an -f mjpeg http://172.17.0.1:1984/api/stream.mjpeg?dst=entrance-mjpeg

The process seems to run fine, but the stream http://imms10:5000/live/webrtc/api/stream.mjpeg?src=entrance-mjpeg no longer works. And ffprobe times out after sending headers. Already tried changing format from mjpeg to mpjpeg or adding -pix_fmt yuv420p but with no difference..

Any idea what may be the problem?

EDIT: You may have doubts about different IP addresses, but they are all interfaces of the same machine and it has already beed double checked.

[AlexxIT]

I don't understand your original idea. Why you using split filter? What is original stream format?

[immek]

I'm a total ffmpeg newbee, so it's quite possible I over engineered something.. . Source stream is a h264 rtsp from a camera. My intention was a pure optimization, I have a dozen different cameras (dahua, reolink, annke, hikvision, h264 and h265), a usb coral and one N100 mini pc. I would like to run the whole thing with detection, recording and addition streams for other clients (tablets etc).

So I wanted to decode the source stream only once and create entrance for detection entrance-full for recording and entrance-mjpeg for wall panel. Using as much hardware acceleration as possible and create as little delay as possible. I thought that split would be the best solution, but maybe I'm wrong :-).

[AlexxIT]

MJPEG won't help you with this. Frigate uses RAW pictures for detections. And will use original format for recording. go2rtc uses original format for streaming. So decoding will be only once for you - detection.

[immek]

I'm using the additional mjpeg stream for sonoff nspanel with homeassistant and webrtc-camera card. After testing different configuration, I'm getting the best results on the panel with mjpeg - not sure why, maybe processing power limitations or weak wifi signal.

[AlexxIT]

streams:
  for_frigate: rtsp://...
  for_nspanel: ffmpeg:for_frigate#video=mjpeg#hardware

[immek]

Now let's get back to the original question 😄 - I would like to understand how to make an incoming mjpeg stream with exec:ffmpeg - why ... -c:v mjpeg -an -f mjpeg http://172.17.0.1:1984/api/stream.mjpeg?dst=entrance-mjpeg from the first post does not work?

I truly admire the capabilities of ffmpeg module, but exec seems more flexibule to me at this time - I can do whatever I want with a stream, change input/output fps, try some magic switches etc.

Also, could you please explain the performance difference between:

streams:
  original_camera_steam: rtsp://...  #let's assume there is only one
  for_frigate-recording: ffmpeg:original_camera_steam#video=h264#hardware#width=...
  for_frigate-detect: ffmpeg:original_camera_steam#video=h264#hardware#width=...
  for_nspanel: ffmpeg:original_camera_steam#video=mjpeg#hardware#width=...

and:

streams:
  original_camera_steam: exec:ffmpeg... split=3... #also hardware
     ... rtsp://127.0.0.1:8554/for_frigate-record
     ... rtsp://127.0.0.1:8554/for_frigate-detect
     ... rtsp://127.0.0.1:8554/for_nspanel
  for_frigate-record: 
  for_frigate-detect:
  for_nspanel:

I thought there would be a major difference between these two, but the true is I don't fully understand how go2rtc works inside, how does the data flow between process pipes and when does encoding/decoding takes place.

[AlexxIT]

You do some strange things in your first post. And you're trying to misuse the software. I think with this way you will only increase the load on the CPU, although you are trying to reduce it.

Frigate will run its own ffmpeg process for raw video decoding.

PS. You using wrong format for incoming source. Check docs.

[immek]

You mean mpjpeg instead of mjpeg? Or something else?

[AlexxIT]

Yes. mpjpeg and mjpeg are different formats

[immek]

Unfortunately, it's something else. I mentioned it in the first post:

Already tried changing format from mjpeg to mpjpeg or adding -pix_fmt yuv420p but with no difference..

Producer (ffmpeg) works, showing an increasing frame counter, but consumers are timing out.

[AlexxIT]

Please let me know if the problem is relevant