search

airockchip / rknn-llm

Other
420 stars 37 forks source link

Guide to adapting new models? #95

Open kinchahoy opened 2 months ago

kinchahoy commented 2 months ago

Folks,

Could you provide a small example or walkthrough on how to adapt a new model type? I'm thinking in particular of multi-modal model types like Florence2ForConditionalGeneration or Phi3VForCausalLM

Thank you!

happyme531 commented 2 months ago

I implemented Florence2: https://huggingface.co/happyme531/Florence-2-base-ft-ONNX-RKNN2/tree/main For Phi3V, it is really hard to do

kinchahoy commented 1 month ago

Thanks for this! Going to try it out. Could you provide a guide or tips on how you did this? I've sort of hacked what you did to try and convert the large FT model, but it's not working great.