search

axinc-ai / ailia-models

The collection of pre-trained, state-of-the-art AI models for ailia SDK
2k stars 318 forks source link

Implement LLaVA #1458

Closed ooe1123 closed 2 months ago

ooe1123 commented 5 months ago

https://github.com/axinc-ai/ailia-models/issues/1411 のPRです。

kyakuno commented 3 months ago

llm -> large_language_modelにrename。

kyakuno commented 3 months ago

モデルアップロード。 https://storage.googleapis.com/ailia-models/llava/embed_tokens.onnx.prototxt など

kyakuno commented 3 months ago

ailia SDKで下記のエラー。

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/ailia/wrapper.py", line 657, in set_input_blob_data
    buf_shape = self.get_blob_shape(idx)
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ailia/wrapper.py", line 491, in get_blob_shape
    core.check_error(code, self.__net)
  File "/usr/local/lib/python3.11/site-packages/ailia/core.py", line 931, in check_error
    raise e(detail)
ailia.core.AiliaUnsettledShapeException: code: -18 (The shape of the blob is not defined.)
+ error detail : Unsettled blob shape.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/kyakuno/Desktop/ailia/ailia-models-ax/large_language_model/llava/llava.py", line 541, in <module>
    main()
  File "/Users/kyakuno/Desktop/ailia/ailia-models-ax/large_language_model/llava/llava.py", line 537, in main
    recognize_from_image(models)
  File "/Users/kyakuno/Desktop/ailia/ailia-models-ax/large_language_model/llava/llava.py", line 492, in recognize_from_image
    output_ids = predict(models, prompt, img)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kyakuno/Desktop/ailia/ailia-models-ax/large_language_model/llava/llava.py", line 453, in predict
    generated_ids = greedy_search(net, inputs_embeds, attention_mask)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/kyakuno/Desktop/ailia/ailia-models-ax/large_language_model/llava/llava.py", line 392, in greedy_search
    logits, past_key_values = forward(
                              ^^^^^^^^
  File "/Users/kyakuno/Desktop/ailia/ailia-models-ax/large_language_model/llava/llava.py", line 147, in forward
    output = net.predict([input_ids, inputs_embeds, position_ids, past_key_values])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/ailia/wrapper.py", line 405, in predict
    self.set_input_blob_data(input[i], idx)
  File "/usr/local/lib/python3.11/site-packages/ailia/wrapper.py", line 661, in set_input_blob_data
    buf_shape = input.shape
                ^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'shape'
kyakuno commented 3 months ago

下記に変更する必要がある。

        input_list = [input_ids, inputs_embeds, position_ids]
        input_list.extend(past_key_values)
        output = net.predict(input_list)
kyakuno commented 3 months ago

CPU推論でもモデルをメモリに乗せるだけで28.51GB使う。 datatype対応してFP16のままメモリに乗せられるようにした方が良さそう。

kyakuno commented 3 months ago

1分以上かかるが推論自体はできた。

kyakuno@mbakk llava % python3 llava.py -e 1
 INFO arg_utils.py (13) : Start!
 INFO arg_utils.py (163) : env_id: 1
 INFO arg_utils.py (166) : CPU-AppleAccelerate
 INFO model_utils.py (89) : ONNX file and Prototxt file are prepared!
 INFO model_utils.py (89) : ONNX file and Prototxt file are prepared!
 INFO model_utils.py (89) : ONNX file and Prototxt file are prepared!
prompt: What are the things I should be cautious about when I visit here?
 INFO llava.py (468) : view.jpg
 INFO llava.py (475) : Start inference...
When visiting this location, which features a pier extending over a large body of water, there are a few things to be cautious about. First, be mindful of the weather conditions, as the pier may be affected by strong winds or storms, which could make it unsafe to walk on. Second, be aware of the water depth and any potential hazards, such as submerged rocks or debris, that could pose a risk to your safety. Additionally, be cautious of the presence of wildlife in the area, as there might be birds or other animals that could pose a threat or disturbance. Finally, be respectful of the environment and other visitors, and follow any posted rules or guidelines to ensure a safe and enjoyable experience for everyone.
 INFO llava.py (503) : Script finished successfully.