dusty-nv
commented
5 months ago Hi @rajeshroy402, please try my suggestion from this post:
https://github.com/dusty-nv/jetson-containers/issues/383#issuecomment-1944269051
Closed rajeshroy402 closed 4 months ago
dusty-nv
commented
5 months ago Hi @rajeshroy402, please try my suggestion from this post:
https://github.com/dusty-nv/jetson-containers/issues/383#issuecomment-1944269051
rajeshroy402
commented
5 months ago Hi @dusty-nv , thank you. Did the changes at line 22 as suggested.
Still not able to run it. I am using the internal storage of the dev kit so for me /mnt/NVME/ wouldn't make sense and I removed that.
Below are my logs.
''' ./run.sh -e SSL_KEY=/data/key.pem -e SSL_CERT=/data/cert.pem -v /mnt/Desktop/jetson-containers/packages/llm/local_llm:/opt/local_llm/local_llm $(./autotag local_llm) python3 -m local_llm.agents.video_query --api=mlc --verbose --model liuhaotian/llava-v1.5-7b --max-new-tokens 32 --video-input /dev/video0 --video-output Display://0 --prompt "How many fingers am I holding up?" Namespace(disable=[''], output='/tmp/autotag', packages=['local_llm'], prefer=['local', 'registry', 'build'], quiet=False, user='dustynv', verbose=False) -- L4T_VERSION=35.2.1 JETPACK_VERSION=5.1 CUDA_VERSION=11.4.315 -- Finding compatible container image for ['local_llm'] dustynv/local_llm:r35.3.1 localuser:root being added to access control list
'''
dusty-nv
commented
5 months ago
./run.sh -e SSL_KEY=/data/key.pem -e SSL_CERT=/data/cert.pem -v /mnt/Desktop/jetson-containers/packages/llm/local_llm:/opt/local_llm/local_llm $(./autotag local_llm)
Error while finding module specification for 'local_llm.agents.video_query' (ModuleNotFoundError: No module named 'local_llm.agents'
Double check that your /mnt/Desktop/jetson-containers/packages/llm/local_llm path is correct - maybe you mean something more like /home/MY_USERNAME/Desktop/jetson-containers/packages/llm/local_llm ?
Also, you can start the container without the python3 -m local_llm.agents.video_query command, and it will just give you an interactive prompt into the container. You can then navigate the container's filesystem and check that the changes you made are reflected inside the container.
rajeshroy402
commented
5 months ago Perfect, thank you.
The display is not opening as it says 'failed to create X11 Window.'
I tried understanding this but couldn't.
Is there another way to fix this?
My command -
./run.sh -e SSL_KEY=/data/key.pem -e SSL_CERT=/data/cert.pem -v /home/rajesh/Desktop/jetson-containers/packages/llm/local_llm:/opt/local_llm/local_llm $(./autotag local_llm) python3 -m local_llm.agents.video_query --api=mlc --verbose --model liuhaotian/llava-v1.5-7b --max-new-tokens 32 --video-input /dev/video0 --video-output display://0 --prompt "How many fingers am I holding up?"
Last lines from my terminal output -
18:03:03 | INFO | using chat template 'llava-v1' for model llava-v1.5-7b
18:03:03 | DEBUG | connected PrintStream to on_eos on channel=0
18:03:03 | DEBUG | connected ChatQuery to PrintStream on channel=0
18:03:03 | DEBUG | processing chat entry 0 role='system' template='${MESSAGE}\n\n' open_user_prompt=False cached=false text='A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.'
18:03:03 | DEBUG | embedding text (1, 32, 4096) float16 -> ```A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.\n\n```
18:03:03 | DEBUG | processing chat entry 1 role='user' template='USER: ${MESSAGE}\n' open_user_prompt=False cached=false text='What is 2+2?'
18:03:03 | DEBUG | embedding text (1, 11, 4096) float16 -> ```USER: What is 2+2?\n```
2+2 is 4.
modprobe: FATAL: Module nvidia not found in directory /lib/modules/5.10.104-tegra
[gstreamer] initialized gstreamer, version 1.16.3.0
[gstreamer] gstCamera -- attempting to create device v4l2:///dev/video0
[gstreamer] gstCamera -- found v4l2 device: C270 HD WEBCAM
[gstreamer] v4l2-proplist, device.path=(string)/dev/video0, udev-probed=(boolean)false, device.api=(string)v4l2, v4l2.device.driver=(string)uvcvideo, v4l2.device.card=(string)"C270\ HD\ WEBCAM", v4l2.device.bus_info=(string)usb-3610000.xhci-4.3, v4l2.device.version=(uint)330344, v4l2.device.capabilities=(uint)2225078273, v4l2.device.device_caps=(uint)69206017;
[gstreamer] gstCamera -- found 36 caps for v4l2 device /dev/video0
[gstreamer] [0] video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 10/1, 15/2, 5/1 };
[gstreamer] [1] video/x-raw, format=(string)YUY2, width=(int)1184, height=(int)656, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 10/1, 15/2, 5/1 };
[gstreamer] [2] video/x-raw, format=(string)YUY2, width=(int)960, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 10/1, 15/2, 5/1 };
[gstreamer] [3] video/x-raw, format=(string)YUY2, width=(int)1024, height=(int)576, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 10/1, 15/2, 5/1 };
[gstreamer] [4] video/x-raw, format=(string)YUY2, width=(int)960, height=(int)544, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [5] video/x-raw, format=(string)YUY2, width=(int)800, height=(int)600, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [6] video/x-raw, format=(string)YUY2, width=(int)864, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [7] video/x-raw, format=(string)YUY2, width=(int)800, height=(int)448, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [8] video/x-raw, format=(string)YUY2, width=(int)752, height=(int)416, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [9] video/x-raw, format=(string)YUY2, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [10] video/x-raw, format=(string)YUY2, width=(int)640, height=(int)360, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [11] video/x-raw, format=(string)YUY2, width=(int)544, height=(int)288, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [12] video/x-raw, format=(string)YUY2, width=(int)432, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [13] video/x-raw, format=(string)YUY2, width=(int)352, height=(int)288, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [14] video/x-raw, format=(string)YUY2, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [15] video/x-raw, format=(string)YUY2, width=(int)320, height=(int)176, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [16] video/x-raw, format=(string)YUY2, width=(int)176, height=(int)144, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [17] video/x-raw, format=(string)YUY2, width=(int)160, height=(int)120, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [18] image/jpeg, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [19] image/jpeg, width=(int)1184, height=(int)656, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [20] image/jpeg, width=(int)960, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [21] image/jpeg, width=(int)1024, height=(int)576, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [22] image/jpeg, width=(int)960, height=(int)544, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [23] image/jpeg, width=(int)800, height=(int)600, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [24] image/jpeg, width=(int)864, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [25] image/jpeg, width=(int)800, height=(int)448, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [26] image/jpeg, width=(int)752, height=(int)416, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [27] image/jpeg, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [28] image/jpeg, width=(int)640, height=(int)360, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [29] image/jpeg, width=(int)544, height=(int)288, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [30] image/jpeg, width=(int)432, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [31] image/jpeg, width=(int)352, height=(int)288, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [32] image/jpeg, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [33] image/jpeg, width=(int)320, height=(int)176, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [34] image/jpeg, width=(int)176, height=(int)144, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] [35] image/jpeg, width=(int)160, height=(int)120, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 30/1, 24/1, 20/1, 15/1, 10/1, 15/2, 5/1 };
[gstreamer] gstCamera -- selected device profile: codec=MJPEG format=unknown width=1280 height=720 framerate=30
[gstreamer] gstCamera pipeline string:
[gstreamer] v4l2src device=/dev/video0 do-timestamp=true ! image/jpeg, width=(int)1280, height=(int)720, framerate=30/1 ! jpegdec name=decoder ! video/x-raw ! appsink name=mysink sync=false
[gstreamer] gstCamera successfully created device v4l2:///dev/video0
[video] created gstCamera from v4l2:///dev/video0
------------------------------------------------
gstCamera video options:
------------------------------------------------
-- URI: v4l2:///dev/video0
- protocol: v4l2
- location: /dev/video0
-- deviceType: v4l2
-- ioType: input
-- codec: MJPEG
-- codecType: cpu
-- width: 1280
-- height: 720
-- frameRate: 30
-- numBuffers: 4
-- zeroCopy: true
-- flipMethod: none
-- sslCert /data/cert.pem
-- sslKey /data/key.pem
------------------------------------------------
[OpenGL] glDisplay -- X screen 0 resolution: 1920x1080
[OpenGL] glDisplay -- X window resolution: 1920x1080
[OpenGL] failed to create X11 Window.
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/opt/local_llm/local_llm/agents/video_query.py", line 117, in <module>
agent = VideoQuery(**vars(args)).run()
File "/opt/local_llm/local_llm/agents/video_query.py", line 40, in __init__
self.video_output = VideoOutput(**kwargs)
File "/opt/local_llm/local_llm/plugins/video.py", line 102, in __init__
self.stream = videoOutput(video_output, options=options)
Exception: jetson.utils -- failed to create videoOutput device
OK, I take it that you have a physical display attached to your Jetson and aren't attempting this over SSH with X11 forwarding right?
This may sound strange, but try changing the code to create the videoOutput node first, above the LLM even.
Also, try running these inside the local_llm container first, to debug the video without the complexity of the LLM:
video-viewer.py /dev/video0 display://0python3 -m local_llm.agents.video_stream \
--video-input /dev/video0 \
--video-output display://0Yes, I have a physical monitor with keyboard and mouse connected directly to the AGX ORIN Dev Kit. I am using terminal inside the same device for this purpose.
wpburns
commented
5 months ago @dusty-nv I'm coincidentally working on trying the get this up and running and having the same problems.
Running:
video-viewer.py /dev/video0 display://0
in the container works fine, however still getting the same failed to create videoOutput device when running video_stream.
Moving the --video-output flag earlier in the command didn't work either.
Thanks in advance.
cahlen
commented
4 months ago I'm at this point also -- everything works perfectly except when trying to display the output video, or save an output file for that matter. I have been successful with --video-input /dev/video0 --video-output webrtc://@:8554/output but nothing ever displays in the OpenGL window.
I spent a lot of time yesterday trying to debug the issue:
$ video-viewer /dev/video0 display://0 works fine on the host (assuming you have jetson-inference installed)./run $(./autotag local_llm) video-viewer /dev/video0 display://0 works correctly../run $(./autotag local_llm) python3 /usr/local/bin/video-viewer.py /dev/video0 display://0 works correctly../run $(./autotag local_llm) python3 -m local_llm.agents.video_stream --video-input /dev/video0 --video-output display://0 --verbose results with a black OpenGL window, however with --verbose I can see that it is capturing frames.There is no difference in the result if instead I place the video-output key before the video-input key:
./run $(./autotag local_llm) python3 -m local_llm.agents.video_stream --video-output display://0 --video-input /dev/video0 --verboseHowever I'm not sure if this lines up with your earlier comment about rearrangement of keys @dusty-nv.
It feels like maybe a callback isn't being run or something? Or the output frame isn't being written, or something like that.
Again, I've only been able to get /dev/video0 to display via webrtc://@:8554/output, but I've never actually been able to see anything on the OpenGL window that pops up by default.
I'd love to help debug this issue if I can be of any help @dusty-nv
dusty-nv
commented
4 months ago @cahlen @wpburns @rajeshroy402 is the common denominator that you are all on JetPack 5?
Over the weekend I tried reproducing this issue on both JetPack 5 and 6 on fresh systems, and unfortunately was unable to.
/dev/video0 --video-output display://0 --verboseresults with a black OpenGL window, however with--verboseI can see that it is capturing frames.
Is the OpenGL window failing to be created, or does it pop-up a window that is literally black? My interpretation is the later.
I'd love to help debug this issue if I can be of any help @dusty-nv
Thanks @cahlen, I appreciate your help, as I've been unable to reproduce this myself. To make it easier for you, if you start the local_llm container by mounting your jetson-container's source tree into the container, you can interactively edit the local_llm source files on your device, and any changes will instantly be reflected inside the container (so you aren't constantly rebuilding it)
./run.sh \
-v /path/to/your/jetson-containers/packages/llm/local_llm:/opt/local_llm/local_llm \
$(./autotag local_llm)Over the weekend, before I even began I re-flashed my Jetson AGX Orin to be completely fresh with Jetpack 6.
The OpenGL window displays, but yes, it is completely black. I've included a few screenshots below with more information. I'll continue to debug on my end also today, and thanks for the mounting hint!
rajeshroy402
commented
4 months ago @cahlen @wpburns @rajeshroy402 is the common denominator that you are all on JetPack 5?
Over the weekend I tried reproducing this issue on both JetPack 5 and 6 on fresh systems, and unfortunately was unable to.
/dev/video0 --video-output display://0 --verboseresults with a black OpenGL window, however with--verboseI can see that it is capturing frames.Is the OpenGL window failing to be created, or does it pop-up a window that is literally black? My interpretation is the later.
I'd love to help debug this issue if I can be of any help @dusty-nv
Thanks @cahlen, I appreciate your help, as I've been unable to reproduce this myself. To make it easier for you, if you start the local_llm container by mounting your jetson-container's source tree into the container, you can interactively edit the local_llm source files on your device, and any changes will instantly be reflected inside the container (so you aren't constantly rebuilding it)
./run.sh \ -v /path/to/your/jetson-containers/packages/llm/local_llm:/opt/local_llm/local_llm \ $(./autotag local_llm)
I am on Jetpack 5.1 while running this.
rajeshroy402
commented
4 months ago I tried webrtc again today and strangely this time it worked.
The only difference is that I changed my root from emmc to sdcard a couple of hours ago. So now, I am booting via the sdcard.
Below is the command that I ran for webRTC:
./run.sh -e SSL_KEY=/data/key.pem -e SSL_CERT=/data/cert.pem -v /home/rajesh/Desktop/jetson-containers/packages/llm/local_llm:/opt/local_llm/local_llm $(./autotag local_llm) python3 -m local_llm.agents.video_query --api=mlc --verbose --model liuhaotian/llava-v1.5-7b --max-new-tokens 32 --video-input /dev/video0 --video-output webrtc://@:8554/output --prompt "How many fingers am I holding up?"I also installed the ssl certificates from here but the video was not playing.
Sharing a screenshot below which contains my terminal logs on the left and webRTC screen at the right.
cahlen
commented
4 months ago That's pretty much where I also ended up getting @rajeshroy402.
With your browser, I ended up using Chrome and disabling any WebRTC stuff in chrome://flags/ and then my video feed appeared.
Nothing in my debugging has really gave me any hint as to why frames won't show up in glDisplay or take in recorded video / save video, especially since I can see that they're being passed to llava because inference comes out correctly within the logs.
@rajeshroy402 are you using a Jetson AGX Orin 32GB also?
rajeshroy402
commented
4 months ago That's pretty much where I also ended up getting @rajeshroy402.
With your browser, I ended up using Chrome and disabling any
WebRTCstuff inchrome://flags/and then my video feed appeared.Nothing in my debugging has really gave me any hint as to why frames won't show up in
glDisplayor take in recorded video / save video, especially since I can see that they're being passed to llava because inference comes out correctly within the logs.@rajeshroy402 are you using a Jetson AGX Orin 32GB also?
@cahlen , Tried setting up with JP6.0 but I only have 128GB sd card (I am booting from a microSD card). So now reinstalling JP5.0.2 Runtime Revision 2 and will run through the entire steps again.
I am using AGX ORIN 64GB Dev Kit. Yesterday night, I was also trying to solve it by editing the python files in utils and llm folders but nothing worked.
@dusty-nv - I think this has something to do with the cuda Render command which is used in VideoOutput. I will try more and see if I can share the logs.
dusty-nv
commented
4 months ago @rajeshroy402 @cahlen okay, I was finally able to reproduce this and track this down - the fix ended up being https://github.com/dusty-nv/jetson-utils/commit/5a5bb422eada7fdbd7d0f53806cf8e0889f2352f
dustynv/local_llm:r36.2.0 has been rebuilt, so pull the latest and please let me know if it fixes the window rendering on your end (the JetPack 5 containers are rebuilding now)
I also attempted to address the mDNS issue in https://github.com/dusty-nv/jetson-containers/commit/87cec8f7f2a4e26d8b793e0805fd47ff6c6152c9 , but .local addresses still don't seem to resolve reliably inside containers, so for now please continue disabling chrome://flags#enable-webrtc-hide-local-ips-with-mdns for WebRTC.
cahlen
commented
4 months ago @dusty-nv That solved it on my end! Thanks for your help!
rajeshroy402
commented
4 months ago @rajeshroy402 @cahlen okay, I was finally able to reproduce this and track this down - the fix ended up being https://github.com/dusty-nv/jetson-utils/commit/5a5bb422eada7fdbd7d0f53806cf8e0889f2352f
dustynv/local_llm:r36.2.0has been rebuilt, so pull the latest and please let me know if it fixes the window rendering on your end (the JetPack 5 containers are rebuilding now)I also attempted to address the mDNS issue in https://github.com/dusty-nv/jetson-containers/commit/87cec8f7f2a4e26d8b793e0805fd47ff6c6152c9 , but
.localaddresses still don't seem to resolve reliably inside containers, so for now please continue disablingchrome://flags/#enable-webrtc-hide-local-ips-with-mdnsfor WebRTC.
Working for me as well.
Thank you.
Head at commit ID- 2d6187b00eaad34a4a51bf1e088baf4a600faa09
Hey Dusty, I am following your tutorial on my new AGX ORIN Dev kit but when I try to run the run.sh file, it fails at the end. I will need your help to solve this, please.
Sharing the terminal output below -