search

airockchip / rknn-llm

Other
347 stars 29 forks source link

Phi-3 error #32

Open dnhkng opened 5 months ago

dnhkng commented 5 months ago

I tried to do a model conversion for the Microsoft Phi-3, and got the following error:

root@443b8f96f39d:/ezrknn-llm/rkllm-toolkit/examples/huggingface# python3 test.py Not support Phi3ForCausalLM! Load model failed!

This model should have a very high priority. Its probably the only model that can run at a good tokens/second on the 3588 and also be useful. It would be a very good showcase for the 3588 and similar systems.

rbrisita commented 5 months ago

I second this! It is a very impressive SLM that passed all my preliminary tests. Curious as to why it failed? @dnhkng did you use the most up-to-date driver?

dnhkng commented 5 months ago

I used the docker instructions from https://github.com/Pelochus/ezrknn-llm

I think this probably needs official support from the rockchip team, as the model is different to Phi-2.

rbrisita commented 5 months ago

Hmm... quickly reviewed it and it doesn't seem to use the newest driver. The docker file should be updated to use this OS:

https://github.com/Pelochus/armbian-build-rknpu-0.9.6

As that removes the 4GB limit which is what I believe is causing your error but all an assumption.

dnhkng commented 5 months ago

The error was in converting the model to RKLLM format, not running the model. The error I posted was in loading the model in the conversation script.

80Builder80 commented 5 months ago

Rockchip should opensource the SDK. There are people ready, willing and able to make it compatible and functional. It seems that this is not a high priority for Rockchip. Let the community help make this a viable product. If Rockchip had LLM functionality that worked as well as the vision models, more people would be willing to purchase boards with their chips.

waydong commented 4 months ago

v1.0.1 has been released, it supports Phi-3.

dnhkng commented 4 months ago

I will soon post a project link that uses this in a very interesting way!

puzzled-pancake commented 4 months ago

I will soon post a project link that uses this in a very interesting way!

Oh I had no idea you wanted to run that on an RK3588 SBC neat. I actually have been looking for an SBC/mini pc to try and mirror what you have done.

av1d commented 4 months ago

I will soon post a project link that uses this in a very interesting way!

Will it run GlaDOS? Would love to see that. I just started digging into it last night (saw you post it on Reddit some weeks ago).

dnhkng commented 4 months ago

That's the goal 😉