search

ankane / onnxruntime-php

Run ONNX models in PHP
MIT License
48 stars 5 forks source link

A question about text generation. Thank you for your amazing work #3

Closed veka-server closed 7 months ago

veka-server commented 8 months ago

Hello Your work is truly impressive. I now generate my tags with wd14 models like SmilingWolf/wd-v1-4-vit-tagger-v2 and it works very well. Thank you. I use it on my RTX3060 and the performance is really decent.

Unfortunately, your tutorial doesn't show how to use text generation (llama, mistral,...). Personally, I'm trying to get Felladrin/onnx-TinyMistral-248M-SFT-v4 to work, but without success so far :(

Could you provide a simple demo for this use case ?

ankane commented 8 months ago

Hi @veka-server, I'd start by looking at the inputs and outputs of the model ($model->inputs() and $model->outputs()). You'll then need to figure out how to transform your data to match, which based on your model, will likely involve digging into the Transformers.js code for pipeline('text-generation', ...).

I did this with GPT-2 a while back in Ruby (by digging into the Transformers Python code), but it'll be different for each model.