./autotag minigpt4 can't start the docker

[ztbupt]

I pulled the docker images and download the models and places them to folders. Because of the network issues, I can not download them by command.

But failed to start the docker. Below is the logs. The hardware is Orin NX 8G. Please help to fix. Thanks!

em3@em3-desktop:~/Downloads/jetson-containers$ ./run.sh $(./autotag minigpt4) /bin/bash -c 'cd /opt/minigpt4.cpp/minigpt4 && python3 webui.py /data/models/huggingface/datasets--maknee--minigpt4-7b-ggml/minigpt4-7B-f16.bin /data/models/huggingface/datasets--maknee--ggml-vicuna-v0-quantized/ggml-vicuna-7B-v0-q5_k.bin' Namespace(disable=[''], output='/tmp/autotag', packages=['minigpt4'], prefer=['local', 'registry', 'build'], quiet=False, user='dustynv', verbose=False) -- L4T_VERSION=35.4.1 JETPACK_VERSION=5.1.2 CUDA_VERSION=11.4.315 -- Finding compatible container image for ['minigpt4'] dustynv/minigpt4:r35.4.1 [sudo] password for em3: MoTTY X11 proxy: No authorisation provided xhost: unable to open display "localhost:10.0"

• docker run --runtime nvidia -it --rm --network host --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /home/em3/Downloads/jetson-containers/data:/data --device /dev/snd --device /dev/bus/usb -e DISPLAY=localhost:10.0 -v /tmp/.X11-unix/:/tmp/.X11-unix -v /tmp/.docker.xauth:/tmp/.docker.xauth -e XAUTHORITY=/tmp/.docker.xauth --device /dev/video0 --device /dev/video1 --device /dev/video2 --device /dev/video3 dustynv/minigpt4:r35.4.1 /bin/bash -c 'cd /opt/minigpt4.cpp/minigpt4 && python3 webui.py /data/models/huggingface/datasets--maknee--minigpt4-7b-ggml/minigpt4-7B-f16.bin /data/models/huggingface/datasets--maknee--ggml-vicuna-v0-quantized/ggml-vicuna-7B-v0-q5_k.bin' ggml_init_cublas: found 1 CUDA devices: Device 0: Orin, compute capability 8.7 llama.cpp: loading model from /data/models/huggingface/datasets--maknee--ggml-vicuna-v0-quantized/ggml-vicuna-7B-v0-q5_k.bin llama_model_load_internal: format = ggjt v3 (latest) llama_model_load_internal: n_vocab = 32001 llama_model_load_internal: n_ctx = 2048 llama_model_load_internal: n_embd = 4096 llama_model_load_internal: n_mult = 256 llama_model_load_internal: n_head = 32 llama_model_load_internal: n_head_kv = 32 llama_model_load_internal: n_layer = 32 llama_model_load_internal: n_rot = 128 llama_model_load_internal: n_gqa = 1 llama_model_load_internal: rnorm_eps = 5.0e-06 llama_model_load_internal: n_ff = 11008 llama_model_load_internal: freq_base = 10000.0 llama_model_load_internal: freq_scale = 1 llama_model_load_internal: ftype = 17 (mostly Q5_K - Medium) llama_model_load_internal: model size = 7B llama_model_load_internal: ggml ctx size = 0.08 MB llama_model_load_internal: using CUDA for GPU acceleration llama_model_load_internal: mem required = 468.40 MB (+ 1024.00 MB per state) llama_model_load_internal: allocating batch_size x (512 kB + n_ctx x 128 B) = 384 MB VRAM for the scratch buffer llama_model_load_internal: offloading 32 repeating layers to GPU llama_model_load_internal: offloading non-repeating layers to GPU llama_model_load_internal: offloading v cache to GPU llama_model_load_internal: offloading k cache to GPU llama_model_load_internal: offloaded 35/35 layers to GPU llama_model_load_internal: total VRAM used: 6031 MB em3@em3-desktop:~/Downloads/jetson-containers$ em3@em3-desktop:~/Downloads/jetson-containers$ em3@em3-desktop:~/Downloads/jetson-containers$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES em3@em3-desktop:~/Downloads/jetson-containers$

[dusty-nv]

@ztbupt I think this stopped working in 8GB memory at some point, also that maknee/minigpt4-7b-ggml model became unmaintained (and GGML format was replaced in llama.cpp by GGUF). Also Mini-GPT4 has been surpassed by other VLMs like Llava, ect.

However, I did just recently get another mini-VLM (NousResearch/Obsidian-3B-V0.5) based on Llava working on Orin Nano 8GB, and am getting it ready to post on Jetson AI Lab!

[ztbupt]

@ztbupt I think this stopped working in 8GB memory at some point, also that maknee/minigpt4-7b-ggml model became unmaintained (and GGML format was replaced in llama.cpp by GGUF). Also Mini-GPT4 has been surpassed by other VLMs like Llava, ect.

However, I did just recently get another mini-VLM (NousResearch/Obsidian-3B-V0.5) based on Llava working on Orin Nano 8GB, and am getting it ready to post on Jetson AI Lab!

Thanks dusty-nv! Which llm contain do you suggest to use on Orin 8G modules? I run text-generation-webui, but looks like work slowly. The minigpt4 is failed due to memory block.

[dusty-nv]

@ztbupt I take it that you have tried Llama-2-7B GGUF on Nano. Using MLC on Orin Nano, for Llama-2-7B I get 16.4 tokens/sec. I am preparing a tutorial page for Jetson AI Lab about Small Language Models (SLMs) that go higher.

[ztbupt]

@ztbupt I take it that you have tried Llama-2-7B GGUF on Nano. Using MLC on Orin Nano, for Llama-2-7B I get 16.4 tokens/sec. I am preparing a tutorial page for Jetson AI Lab about Small Language Models (SLMs) that go higher.

Got it. Looking forward your tutorial page.