search

e-p-armstrong / augmentoolkit

Convert Compute And Books Into Instruct-Tuning Datasets
MIT License
585 stars 79 forks source link

Dataset generation for TGI 1.2.0 #7

Closed pallavijaini0525 closed 3 months ago

pallavijaini0525 commented 3 months ago

Thank you for this amazing tool.

I intend to utilize this tool to transform my PDF data into a fine-tuning dataset for LLAMA-7B. Presently, when utilizing TGI version 1.2.0 for the inference API, the processing.py fails saying /chat/completions endpoint is missing. TGI 1.2.0 does not support open AI message API. Is there a way to adapt the code to function with alternative inference APIs such as torchserve or with TGI 2.0?

e-p-armstrong commented 3 months ago

Yeah sure, you can do that. you'll want to look into the submit_chat and submit_completion methods in the EngineWrapper class in engine_wrapperclass.py, and also change the init method of that class to, if mode=="torchserve" or something, set the object's client object to an API object for the specific API you want instead of AsyncOpenAI image When you're done with the init method that's when you'll go into the submit methods on EngineWrapper and modify them to work with whatever API library you're using by adding an if statement with if self.mode=="torchserve" and then the logic you need.

Then you go into config.yaml and set MODE to torchserve and you SHOULD be good to go unless I missed something. Let me know if it works! And actually if you manage to do it without breaking other functionality I'd welcome the changes as a PR to support Torchserve

e-p-armstrong commented 3 months ago

Closing because I think this problem has been resolved