[FR] h264 Output for ustreamer

[boehser-enkel]

Hi,

i was excited to see obico support and installed it locally. Linking the printer worked but i only get webcam snapshots instead of streams.

I saw https://www.obico.io/docs/user-guides/webcam-stream-stuck-at-1-10-fps/#2-is-octoprint-running-on-a-raspberry-pi: If you have installed docker on the Raspberry Pi, and are running OctoPrint Or Klipper/Moonraker inside the docker containers, you won't get the webcam stream either.

Is this the case?

Moonraker-obico logs:

INFO  backoff - Backing off wait_for_janus(...) for 0.0s (ConnectionRefusedError: [Errno 111] Connection refused)
2023-09-06 06:09:55,964      INFO  backoff - Backing off wait_for_janus(...) for 1.2s (ConnectionRefusedError: [Errno 111] Connection refused)
2023-09-06 06:09:58,178      INFO  backoff - Backing off wait_for_janus(...) for 1.9s (ConnectionRefusedError: [Errno 111] Connection refused)
2023-09-06 06:10:01,112      INFO  backoff - Backing off wait_for_janus(...) for 3.0s (ConnectionRefusedError: [Errno 111] Connection refused)
2023-09-06 06:10:05,120      INFO  backoff - Backing off wait_for_janus(...) for 3.1s (ConnectionRefusedError: [Errno 111] Connection refused)
2023-09-06 06:10:09,190      INFO  backoff - Backing off wait_for_janus(...) for 16.2s (ConnectionRefusedError: [Errno 111] Connection refused)
2023-09-06 06:10:14,494      INFO  backoff - Backing off capture_jpeg(...) for 0.2s (requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /webcam/?action=snapshot (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7a24a10>: Failed to establish a new connection: [Errno 111] Connection refused')))
2023-09-06 06:10:14,675      INFO  backoff - Backing off capture_jpeg(...) for 1.8s (requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /webcam/?action=snapshot (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7a24490>: Failed to establish a new connection: [Errno 111] Connection refused')))
2023-09-06 06:10:16,452     ERROR  backoff - Giving up capture_jpeg(...) after 3 tries (requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /webcam/?action=snapshot (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7a57dd0>: Failed to establish a new connection: [Errno 111] Connection refused')))
2023-09-06 06:10:16,452   WARNING  obico.webcam_capture - Failed to capture jpeg - HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /webcam/?action=snapshot (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7a57dd0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2023-09-06 06:10:16,458      INFO  backoff - Backing off capture_jpeg(...) for 0.2s (requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /webcam/?action=snapshot (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7a89b10>: Failed to establish a new connection: [Errno 111] Connection refused')))
2023-09-06 06:10:16,660      INFO  backoff - Backing off capture_jpeg(...) for 0.9s (requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /webcam/?action=snapshot (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7a8a8d0>: Failed to establish a new connection: [Errno 111] Connection refused')))
2023-09-06 06:10:17,562     ERROR  backoff - Giving up capture_jpeg(...) after 3 tries (requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /webcam/?action=snapshot (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa7a95b90>: Failed to establish a new connection: [Errno 111] Connection refused')))

greetings

[mkuf]

Hi,

could you clarify if you do not see an image at all or if you only see snapshots of your webcam?

From the logs I see two potential issues.

1. Janus is missing
2. The webcam url in the config file seems to be not set correctly

For 1 I created a PR to add janus to the image https://github.com/TheSpaghettiDetective/moonraker-obico/pull/78, as this may be related. For 2, please share your config/moonraker-obico.cfg and describe your Webcam setup. I recall you're not using the webcam service from prind, right?

When those issues are fixed, we can try to configure the moonraker-obico container in a way so that the h264 encoder of the Pi mentioned in the docs you linked may be used.

-Markus

[boehser-enkel]

I only see snapshots

It's a usb cam on my pi which i use in with prind as normal i think. `[server] url = http://192.168.88.202:3334 auth_token = 2aa50802741c0659XXXX

[moonraker] host = moonraker port = 7125

[webcam] disable_video_streaming = False snapshot_url = http://webcam:8080/snapshot stream_url = http://webcam:8080/stream

[logging] path = /opt/printer_data/logs/moonraker-obico.log level = INFO

[tunnel] dest_host = traefik dest_port = 80 dest_is_ssl = False

`

[mkuf]

The PR for 1 was merged, please pull the latest moonraker-obico image onto your machine and add the following to your docker-compose.override.yaml:

services:
  moonraker-obico:
    privileged: true

After the stack is back up again, please share the full log of the moonraker-obico service.

[boehser-enkel]

` ext_info = self.server_conn.send_http_request('GET', f'/ent/api/printers/{self.printer_id}/ext/', timeout=60, raise_exception=True)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/opt/moonraker-obico/moonraker_obico/server_conn.py", line 150, in send_http_request

resp.raise_for_status()

File "/opt/venv/lib/python3.11/site-packages/requests/models.py", line 953, in raise_for_status

raise HTTPError(http_error_msg, response=self)

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://192.168.88.202:3334/ent/api/printers/2/ext/

2023-09-11 05:55:44,687 WARNING obico.moonraker_conn - api key is unset, trying to fetch one

2023-09-11 05:55:44,692 INFO obico.janus - Starting webcam streamer

2023-09-11 05:55:44,697 INFO obico.webcam_stream - Trying to start ffmpeg using camera-streamer H.264 source

2023-09-11 05:55:44,711 INFO obico.moonraker_conn - connection is open

2023-09-11 05:55:44,937 INFO obico.moonraker_conn - Klippy ready

2023-09-11 05:55:44,959 WARNING obico.app - error response from moonraker, Event(name='message', data={'jsonrpc': '2.0', 'error': {'code': 400, 'message': "{'error': 'WebRequestError', 'message': 'Invalid argument'}"}, 'id': 30098}, sender='moonrakerconn')

2023-09-11 05:55:45,166 ERROR obico.webcam_stream - STDOUT:

None

STDERR:

b'[tcp @ 0x555c7374f0] Connection to tcp://127.0.0.1:8080 failed: Connection refused\nhttp://127.0.0.1:8080/video.mp4: Connection refused\n'

2023-09-11 05:55:45,167 INFO obico.webcam_stream - No camera-stream H.264 source found. Continue to legacy streaming: ffmpeg failed! Exit code: 1

2023-09-11 05:55:45,179 INFO obico.app - detected state change: Offline -> Operational

2023-09-11 05:55:45,957 ERROR obico.utils -

Traceback (most recent call last):

File "/opt/moonraker-obico/moonraker_obico/webcam_stream.py", line 83, in video_pipeline

self.ffmpeg_from_mjpeg()

File "/opt/moonraker-obico/moonraker_obico/webcam_stream.py", line 130, in ffmpeg_from_mjpeg

encoder = h264_encoder()

          ^^^^^^^^^^^^^^

File "/opt/moonraker-obico/moonraker_obico/webcam_stream.py", line 114, in h264_encoder

raise Exception('No ffmpeg found, or ffmpeg does NOT support h264_omx/h264_v4l2m2m encoding.')

Exception: No ffmpeg found, or ffmpeg does NOT support h264_omx/h264_v4l2m2m encoding.`

The stream tries to load but i only get snapshot updates i think (on reload)

[mkuf]

1. Please format the log in a way that it is more readable, in the current form I'm having a hard time identifying the actual error message.
2. Please run script/get-info.sh and upload the generated archive here
3. Also execute the following command on your machine and share the output

   docker run --rm -it --entrypoint ffmpeg ghcr.io/thespaghettidetective/moonraker-obico:latest -codecs | grep -i h264

[boehser-enkel]

Log: https://pastebin.com/uqKvHe7y Info: https://www.dropbox.com/scl/fi/hjuca5rtqwmy87z1fp5gw/prind-info-07062023-080800.tar.gz?rlkey=h4d8iume9ubgvn093isrbrc10&dl=0 DEV.LS h264 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264 h264_v4l2m2m h264_cuvid ) (encoders: libx264 libx264rgb h264_nvenc h264_omx h264_v4l2m2m h264_vaapi )

[mkuf]

That archive seems to be from 07.06.2023 and does not contain any traces of moonraker-obico.
Make sure your copy of prind is up to date (=v1.9.0), create a new archive and share it here.

[boehser-enkel]

Oh yes i got the wrong one: https://www.dropbox.com/scl/fi/opz9qklra3qs9voi0ewa3/prind-info-14092023-141453.tar.gz?rlkey=vadtyl3ppcy14bm2rycbkmvbj&dl=0

[mkuf]

2023-09-14 12:10:10,593      INFO  obico.app - starting moonraker-obico (v1.4.12)
2023-09-14 12:10:10,593      INFO  obico.app - Fetching linked printer...
2023-09-14 12:10:10,772      INFO  obico.app - Linked printer: {'is_pro': True, 'id': 2, 'name': 'Elegoo Neptune 3 Pro'}
2023-09-14 12:10:10,936     ERROR  obico.nozzlecam - Failed to build nozzle config
Traceback (most recent call last):
  File "/opt/moonraker-obico/moonraker_obico/nozzlecam.py", line 55, in create_nozzlecam_config
    ext_info = self.server_conn.send_http_request('GET', f'/ent/api/printers/{self.printer_id}/ext/', timeout=60, raise_exception=True)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/moonraker-obico/moonraker_obico/server_conn.py", line 150, in send_http_request
    resp.raise_for_status()
  File "/opt/venv/lib/python3.11/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://[redacted]/ent/api/printers/2/ext/
2023-09-14 12:10:10,980   WARNING  obico.moonraker_conn - api key is unset, trying to fetch one
2023-09-14 12:10:10,986      INFO  obico.janus - Starting webcam streamer
2023-09-14 12:10:10,989      INFO  obico.webcam_stream - Trying to start ffmpeg using camera-streamer H.264 source
2023-09-14 12:10:11,027      INFO  obico.moonraker_conn - connection is open
2023-09-14 12:10:11,044      INFO  backoff - Backing off wait_for_klippy_ready(...) for 1.0s (False)
2023-09-14 12:10:11,431     ERROR  obico.webcam_stream - STDOUT:
None
STDERR:
b'[tcp @ 0x55af1624f0] Connection to tcp://127.0.0.1:8080 failed: Connection refused\nhttp://127.0.0.1:8080/video.mp4: Connection refused\n'

2023-09-14 12:10:11,432      INFO  obico.webcam_stream - No camera-stream H.264 source found. Continue to legacy streaming: ffmpeg failed! Exit code: 1
2023-09-14 12:10:12,138      INFO  obico.moonraker_conn - Klippy ready
2023-09-14 12:10:12,259     ERROR  obico.utils - 
Traceback (most recent call last):
  File "/opt/moonraker-obico/moonraker_obico/webcam_stream.py", line 83, in video_pipeline
    self.ffmpeg_from_mjpeg()
  File "/opt/moonraker-obico/moonraker_obico/webcam_stream.py", line 130, in ffmpeg_from_mjpeg
    encoder = h264_encoder()
              ^^^^^^^^^^^^^^
  File "/opt/moonraker-obico/moonraker_obico/webcam_stream.py", line 114, in h264_encoder
    raise Exception('No ffmpeg found, or ffmpeg does NOT support h264_omx/h264_v4l2m2m encoding.')
Exception: No ffmpeg found, or ffmpeg does NOT support h264_omx/h264_v4l2m2m encoding.
2023-09-14 12:10:12,392      INFO  obico.app - detected state change: Offline -> Operational

This is the relevant part of the log.
I'm not really sure what is happening here, maybe @kennethjiang can give some more insights on this.

What I can say is, that the service does have access to the PIs devices, as it is run in privileged mode.
Furthermore, ffmpeg is available and the mentioned codecs are also available as this output from https://github.com/mkuf/prind/issues/93#issuecomment-1719339075 shows:

DEV.LS h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264 h264_v4l2m2m h264_cuvid ) (encoders: libx264 libx264rgb h264_nvenc h264_omx h264_v4l2m2m h264_vaapi )

[kennethjiang]

@mkuf Will take a look later today.

[kennethjiang]

@boehser-enkel The statement in the doc doesn't apply here as it seems that the HW accelerator is available to moonraker-obico.

Can you let me know the steps you took to set up printd on your Raspberry Pi? I'd like to reproduce the issue by following the exact steps.

[mkuf]

From the support archive @boehser-enkel shared, the following compose file should be sufficient for reproducing the behaviour on a Raspberry Pi.

Their config file is in line with the basic file provided by prind.
For debugging/reproducing you'd probably want to update the moonraker host and port w/ the data of a printer available to you.

services:
  moonraker-obico:
    image: ghcr.io/thespaghettidetective/moonraker-obico:latest
    restart: unless-stopped
    privileged: true
    volumes:
      - ./config:/opt/printer_data/config
      - log:/opt/printer_data/logs

  webcam:
    image: mkuf/ustreamer:latest
    restart: unless-stopped
    command: --host=0.0.0.0 --port=8080 --slowdown --device=/dev/webcam --resolution=1280x960 --format=MJPEG --desired-fps=30
    devices:
      - /dev/video0:/dev/webcam

[kennethjiang]

Here is what I have found so far @mkuf. I ran

ffmpeg -re -v debug -i /opt/moonraker-obico/moonraker_obico/bin/ffmpeg/test-video.mp4 -pix_fmt yuv420p -vcodec h264_v4l2m2m -an -f rtp 'rtp://localhost:8014?pkt_size=1300' in the moonraker-obico container, got this output:

...
[auto_scale_0 @ 0x2b1d280] w:640 h:480 fmt:yuvj420p sar:0/1 -> w:640 h:480 fmt:yuv420p sar:0/1 flags:0x0
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video31
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video23
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video22
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video21
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video20
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video19
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video18
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video16
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video15
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video14
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video13
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video12
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video11
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video10
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video1
[h264_v4l2m2m @ 0x2a70f50] probing device /dev/video0
[h264_v4l2m2m @ 0x2a70f50] Could not find a valid device
[h264_v4l2m2m @ 0x2a70f50] can't configure encoder
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[AVIOContext @ 0x2a51430] Statistics: 0 bytes written, 0 seeks, 0 writeouts
[AVIOContext @ 0x2a516f0] Statistics: 26482 bytes read, 0 seeks
Conversion failed!

This means that the encoder device was not correctly mapped from the host into the container.

If you have ideas why this is the case, please let me know. Otherwise, I'll dig a bit deeper.

[kennethjiang]

One more piece of info: if I launched a container with docker run --privileged --rm -it balenalib/raspberry-pi bash, this command will work. This means the encoder device is correctly mapped into the container.

I guess now we need to figure out the difference between these 2 situations.

BTW I'd add:

services:
  moonraker-obico:
    privileged: true

to docker-compose.yml or docker-compose.override.yml.

[mkuf]

That snippet I created earlier was for the wrong service (mobileraker_companion instead of moonraker-obico :face_in_clouds: ). I just edited it to reflect the correct one.

I could imagine that the obico user does not have access to the video devices, as their access is limited to root:video.
For testing, you can set user: 0:0 for the container to run as root.

The override file should look like this:

services:
  moonraker-obico:
    privileged: true
    user: 0:0

Alternatively, if you're testing outside of prind, the a full service definition:

services:
  moonraker-obico:
    image: ghcr.io/thespaghettidetective/moonraker-obico:latest
    restart: unless-stopped
    volumes:
      - ./config:/opt/printer_data/config
      - log:/opt/printer_data/logs
    privileged: true
    user: 0:0

[kennethjiang]

I tried:

services:
  moonraker-obico:
    privileged: true
    user: 0:0

And it worked. @boehser-enkel Can you try it to see if it also solves the problem for you?

[mkuf]

That is great to hear.
Could you also give this next snippet a try @kennethjiang? I do not own a Pi, where I could verify that change.

This adds the obico user to the video group inside the container and should make it possible to run the process as the unprivileged user.
If that succeeds for you, I'll prepare a PR to permanently add this to the moonraker-obico Image.

services:
  moonraker-obico:
    privileged: true
    build:
      dockerfile_inline: |
        FROM ghcr.io/thespaghettidetective/moonraker-obico:latest
        USER root
        RUN usermod obico --append --groups video
        USER obico

[boehser-enkel]

I tried:

services:
  moonraker-obico:
    privileged: true
    user: 0:0

And it worked. @boehser-enkel Can you try it to see if it also solves the problem for you?

Unfortunately it does not:

b'[tcp @ 0x55844a14f0] Connection to tcp://127.0.0.1:8080 failed: Connection refused\nhttp://127.0.0.1:8080/video.mp4: Connection refused\n'
2023-09-20 05:36:41,139      INFO  obico.webcam_stream - No camera-stream H.264 source found. Continue to legacy streaming: ffmpeg failed! Exit code: 1

[kennethjiang]

I tried:

services:
  moonraker-obico:
    privileged: true
    user: 0:0

And it worked. @boehser-enkel Can you try it to see if it also solves the problem for you?

Unfortunately it does not:

b'[tcp @ 0x55844a14f0] Connection to tcp://127.0.0.1:8080 failed: Connection refused\nhttp://127.0.0.1:8080/video.mp4: Connection refused\n'
2023-09-20 05:36:41,139      INFO  obico.webcam_stream - No camera-stream H.264 source found. Continue to legacy streaming: ffmpeg failed! Exit code: 1

This is expected. This is to test if the service is camera-streamer. Since printd is using ustreamer, http://127.0.0.1:8080/video.mp4 is expected to fail.

Do you see other errors?

[kennethjiang]

That is great to hear. Could you also give this next snippet a try @kennethjiang? I do not own a Pi, where I could verify that change.

This adds the obico user to the video group inside the container and should make it possible to run the process as the unprivileged user. If that succeeds for you, I'll prepare a PR to permanently add this to the moonraker-obico Image.

services:
  moonraker-obico:
    privileged: true
    build:
      dockerfile_inline: |
        FROM ghcr.io/thespaghettidetective/moonraker-obico:latest
        USER root
        RUN usermod obico --append --groups video
        USER obico

@mkuf Sorry for taking so long. I just confirmed that this worked on my RPi! I guess you can commit this change to your code.

[mkuf]

Thanks for testing @kennethjiang 🙇

Privileged mode has been added to the main compose file in https://github.com/mkuf/prind/commit/68a57f9aaefbcd0cb20fbbcd772557462ac422bc
A PR to add the obico user to the video group within the docker image has also been created to make this available for everyone that is using the image: https://github.com/TheSpaghettiDetective/moonraker-obico/pull/79

With these changes it is possible to use a Pis Video encoder without any extra config necessary.

[kennethjiang]

Awesome! PR merged!

[mkuf]

Thanks!

With the PR merged and prind v1.9.1 released, this should now work for anyone.

[mkuf]

@boehser-enkel do you have any further logs, as requested by @kennethjiang?
If not, I'd consider this issue resolved.

[boehser-enkel]

@mkuf @kennethjiang logs: https://pastebin.com/NNNQ8ex4

[kennethjiang]

@mkuf @kennethjiang logs: https://pastebin.com/NNNQ8ex4

This error shouldn't matter. Please notice:

... No camera-stream H.264 source found. Continue to legacy streaming...

[mkuf]

It should be possible to enable h264 streaming for ustreamer, based on the official docs: https://github.com/pikvm/ustreamer/blob/master/docs/h264.md

I'll have to do some testing if this setup can be added to the ustreamer image so h264 is supported by default.

Until then you may be better off using a webcam streamer that supports h264 out of the box and install it on your host @boehser-enkel, but you'll have to figure that out yourself.

[mkuf]

As h264 streaming/encoding is dependent on the available hardware and platform, this would result in a multitude of different image versions to support any possible combination out there, thats why I decided to not further invest time in adding a h264 streamer to the project.
That said, I'm happy to accept Pull Requests that add this functionality.

I suggest to follow the instructions at https://github.com/ayufan/camera-streamer to get h264 streaming up and running on your host if that is a requirement for anyones setup.