search

pytorch / torchchat

Run PyTorch LLMs locally on servers, desktop and mobile
BSD 3-Clause "New" or "Revised" License
3.35k stars 219 forks source link

How to deploy a new model by torchchat? #1038

Open liu8060 opened 2 months ago

liu8060 commented 2 months ago

I want to use torchchat to load the trained model directly from the local. How to change the torchchat/config/data/models.json? Need to change download and convert in download.py?And, what other documents may need to be changed?

Jack-Khuu commented 2 months ago

Glad to have you try things out

What file format is the local model you're working with?

liu8060 commented 2 months ago

two formats One is a folder format, which includes the model’s weight files, tokenizer files, and other necessary components. The other format is a PyTorch .pth file, which contains the entire model.

The model's base model is llama3-8b

Jack-Khuu commented 2 months ago

If the model is accessible from huggingface: Here's an example PR of how you can add it

https://github.com/pytorch/torchchat/pull/947 Specifically the known_model_configs and model.json

The other format is a PyTorch .pth file, which contains the entire model.

For this you can can add a known_model_config (based on your params.json with "use_tiktoken"=true for llama3 derivatives) and then point to your pth with --checkpoint-path

For example: python torchchat.py generate --checkpoint-path ~/Desktop/test/consolidated.00.pth --prompt "What should I do in Manhattan Chinatown?" --device mps --num-samples 3

Jack-Khuu commented 2 months ago

One is a folder format, which includes the model’s weight files, tokenizer files, and other necessary components.

Similar to this case? https://github.com/pytorch/torchchat/issues/1040