Closed tfuru-macnica closed 10 months ago
dusty-nv
commented
10 months ago
minigpt4:r35.4.1-onnx
@tfuru-macnica I think because you had attempted a build of minigpt4 container at some point, hence it is prioritizing your local images. If you either completed the build, or removed the failed build containers, then autotag would resume working as expected. I will make a note to incorporate greater checking into autotag.
tfuru-macnica
commented
10 months ago @dusty-nv Thank you for your reply. I just removed the dustynv/minigpt4:r35.4.1 image from my Jetson and issued autotag command. But the issue was still there.
$ ./autotag minigpt4
Namespace(disable=[''], output='/tmp/autotag', packages=['minigpt4'], prefer=['local', 'registry', 'build'], quiet=False, user='dustynv', verbose=False)
-- L4T_VERSION=35.4.1 JETPACK_VERSION=5.1.2 CUDA_VERSION=11.4.315
-- Finding compatible container image for ['minigpt4']
Found compatible container dustynv/minigpt4:r35.3.1 (2023-11-05, 5.9GB) - would you like to pull it? [Y/n] Do I need to remove all images from my Jetson?
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
text_generation_webui_asr latest bcc7fb41c723 7 days ago 14.5GB
dustynv/stable-diffusion-webui r35.3.1 6675881993a3 8 days ago 16GB
dustynv/text-generation-webui r35.4.1 15e0e768078e 5 weeks ago 14.3GB
dustynv/riva-client python-r35.4.1 33c8d382d631 5 weeks ago 9.86GB
nvcr.io/nvidia/riva/riva-speech 2.13.0-l4t-aarch64 4569877da686 2 months ago 12.4GB
nvcr.io/nvidia/l4t-jetpack r35.4.1 5c923ac521a3 3 months ago 9.71GBThanks.
dusty-nv
commented
10 months ago @tfuru-macnica it's because dustynv/minigpt4:r35.3.1 was uploaded more recently and should be compatible with r35.4.1. Does it not work for you?
EDIT: I see in your original post, you said it fails to start the app when using r35.3.1 image. What error do you get?
tfuru-macnica
commented
10 months ago @dusty-nv Here is the error when autotag selects dustynv/minigpt4:r35.3.1 as compatible image:
$ ./run.sh $(./autotag minigpt4) /bin/bash -c 'cd /opt/minigpt4.cpp/minigpt4 && python3 webui.py \
> $(huggingface-downloader --type=dataset maknee/minigpt4-13b-ggml/minigpt4-13B-f16.bin) \
> $(huggingface-downloader --type=dataset maknee/ggml-vicuna-v0-quantized/ggml-vicuna-13B-v0-q5_k.bin)'
Namespace(disable=[''], output='/tmp/autotag', packages=['minigpt4'], prefer=['local', 'registry', 'build'], quiet=False, user='dustynv', verbose=False)
-- L4T_VERSION=35.4.1 JETPACK_VERSION=5.1.2 CUDA_VERSION=11.4.315
-- Finding compatible container image for ['minigpt4']
Found compatible container dustynv/minigpt4:r35.3.1 (2023-11-05, 5.9GB) - would you like to pull it? [Y/n]
dustynv/minigpt4:r35.3.1
localuser:root being added to access control list
xauth: file /tmp/.docker.xauth does not exist
+ docker run --runtime nvidia -it --rm --network host --volume /tmp/argus_socket:/tmp/argus_socket --volume /etc/enctune.conf:/etc/enctune.conf --volume /etc/nv_tegra_release:/etc/nv_tegra_release --volume /tmp/nv_jetson_model:/tmp/nv_jetson_model --volume /home/jetson/jetson-containers/data:/data --device /dev/snd --device /dev/bus/usb -e DISPLAY=:0 -v /tmp/.X11-unix/:/tmp/.X11-unix -v /tmp/.docker.xauth:/tmp/.docker.xauth -e XAUTHORITY=/tmp/.docker.xauth dustynv/minigpt4:r35.3.1 /bin/bash -c 'cd /opt/minigpt4.cpp/minigpt4 && python3 webui.py \
$(huggingface-downloader --type=dataset maknee/minigpt4-13b-ggml/minigpt4-13B-f16.bin) \
$(huggingface-downloader --type=dataset maknee/ggml-vicuna-v0-quantized/ggml-vicuna-13B-v0-q5_k.bin)'
Unable to find image 'dustynv/minigpt4:r35.3.1' locally
r35.3.1: Pulling from dustynv/minigpt4
970e18d4d6e7: Already exists
dd5b4c4a0ca1: Already exists
f633ec2f194a: Already exists
89174908b582: Already exists
136a1cfda643: Already exists
b5b63519926f: Already exists
f5091be6d46f: Already exists
9fa2b9fb025c: Already exists
4a603e9470a6: Already exists
b9bf6a5cabd4: Already exists
e13ae8fc5f8d: Already exists
4903681b8404: Already exists
b72c80d5d95d: Already exists
5c85151b5936: Already exists
1e1721532d25: Already exists
65193e0b82ac: Already exists
dab88721977e: Already exists
784e2d3fe8c2: Already exists
0d37924a7cbb: Already exists
1d915aaf7512: Already exists
fa2481f6ed6f: Already exists
2081645ab80e: Already exists
cec7184c25a0: Already exists
84423a00ce60: Already exists
61773d2d0b6c: Already exists
9be0c23b58d7: Already exists
d6cfe730c2c3: Already exists
2833cccafb32: Already exists
2a0cd3da88ca: Already exists
7e5e252e963c: Already exists
99d922913789: Already exists
d8027f59b837: Already exists
85b86d2988f1: Already exists
c7efeffb80dc: Already exists
87fbaf83c80c: Already exists
b9b65d6b37ce: Already exists
0eb4a58e25a9: Already exists
7eeaed4f68a0: Already exists
3bc5a309faa4: Already exists
3b0c612ea35f: Already exists
3e3178f6652c: Already exists
5e9d60facea8: Already exists
e51f686a8e9e: Already exists
ebbaf669c46d: Already exists
5ee91f9b3d7b: Already exists
3df68dbb84aa: Already exists
9cb641679dbb: Already exists
91e9dd029a94: Already exists
1c4993c392e1: Already exists
368983b3a52a: Already exists
8a012f2dcab1: Already exists
94140308d863: Pull complete
47df864b25bf: Pull complete
087a80c2dc0e: Pull complete
667fdd2182cf: Pull complete
63f949dcae9c: Pull complete
1e4c5e18d7c6: Pull complete
5ac0825beef0: Pull complete
Digest: sha256:0663d321f8a566dcc04837923bff02591ad2ebb15d93ee353552178502ea44a7
Status: Downloaded newer image for dustynv/minigpt4:r35.3.1
Downloading maknee/minigpt4-13b-ggml/minigpt4-13B-f16.bin to /data/models/huggingface
repo_id maknee/minigpt4-13b-ggml
filename minigpt4-13B-f16.bin
Downloaded maknee/minigpt4-13b-ggml/minigpt4-13B-f16.bin to: /data/models/huggingface/datasets--maknee--minigpt4-13b-ggml/snapshots/08e9f5ce36f6c1c3a2cc1abf9872bb99fb20c4e4/minigpt4-13B-f16.bin
Downloading maknee/ggml-vicuna-v0-quantized/ggml-vicuna-13B-v0-q5_k.bin to /data/models/huggingface
repo_id maknee/ggml-vicuna-v0-quantized
filename ggml-vicuna-13B-v0-q5_k.bin
Downloaded maknee/ggml-vicuna-v0-quantized/ggml-vicuna-13B-v0-q5_k.bin to: /data/models/huggingface/datasets--maknee--ggml-vicuna-v0-quantized/snapshots/1d8789f34eb803bf52daf895c7ecfd2559cf5ccc/ggml-vicuna-13B-v0-q5_k.bin
ggml_init_cublas: found 1 CUDA devices:
Device 0: Orin, compute capability 8.7
llama.cpp: loading model from /data/models/huggingface/datasets--maknee--ggml-vicuna-v0-quantized/snapshots/1d8789f34eb803bf52daf895c7ecfd2559cf5ccc/ggml-vicuna-13B-v0-q5_k.bin
llama_model_load_internal: format = ggjt v3 (latest)
llama_model_load_internal: n_vocab = 32001
llama_model_load_internal: n_ctx = 2048
llama_model_load_internal: n_embd = 5120
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 40
llama_model_load_internal: n_head_kv = 40
llama_model_load_internal: n_layer = 40
llama_model_load_internal: n_rot = 128
llama_model_load_internal: n_gqa = 1
llama_model_load_internal: rnorm_eps = 5.0e-06
llama_model_load_internal: n_ff = 13824
llama_model_load_internal: freq_base = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype = 17 (mostly Q5_K - Medium)
llama_model_load_internal: model size = 13B
llama_model_load_internal: ggml ctx size = 0.11 MB
llama_model_load_internal: using CUDA for GPU acceleration
llama_model_load_internal: mem required = 582.00 MB (+ 1600.00 MB per state)
llama_model_load_internal: allocating batch_size x (640 kB + n_ctx x 160 B) = 480 MB VRAM for the scratch buffer
llama_model_load_internal: offloading 40 repeating layers to GPU
llama_model_load_internal: offloading non-repeating layers to GPU
llama_model_load_internal: offloading v cache to GPU
llama_model_load_internal: offloading k cache to GPU
llama_model_load_internal: offloaded 43/43 layers to GPU
llama_model_load_internal: total VRAM used: 10959 MB
llama_new_context_with_model: kv self size = 1600.00 MB
/usr/local/lib/python3.8/dist-packages/gradio/layouts/column.py:55: UserWarning: 'scale' value should be an integer. Using 0.5 will cause issues.
warnings.warn(
Traceback (most recent call last):
File "webui.py", line 129, in <module>
start(share_link)
File "webui.py", line 87, in start
submit = gr.Button(value="Send message", variant="secondary").style(full_width=True)
AttributeError: 'Button' object has no attribute 'style'
submit = gr.Button(value="Send message", variant="secondary").style(full_width=True) AttributeError: 'Button' object has no attribute 'style'
Aha, okay thanks - the Gradio version needed pinned. I did that in https://github.com/dusty-nv/jetson-containers/commit/f7c7ef4ce5fb984c6525e4f3851a9de6ac04151c and rebuilt the containers. If you do sudo docker pull dustynv/minigpt4:r35.3.1, it should work now. Rebuilt minigpt4 container tags are updated here:
tfuru-macnica
commented
10 months ago @dusty-nv Thank you very much! It has worked fine.
On Jetson AGX Orin Dev Kit (32GB) with L4T 35.4.1,
So the autotag will select the dustynv/minigpt4:r35.3.1 image as a compatible docker image. However the dustynv/minigpt4:r35.3.1 will fail to start the MiniGPT-4 application.
If you specify the tag dustynv/minigpt4:r35.4.1 manually as follows, the application will be launched fine.