search

mzbac / AutoGPTQ-API

Host the GPTQ model using AutoGPTQ as an API that is compatible with text generation UI API.
91 stars 14 forks source link

Run on Google Colab #1

Closed apcameron closed 1 year ago

apcameron commented 1 year ago

What would need to be done to get this running on Google Colab so you can connect to it remotely?

ashercn97 commented 1 year ago

Yes!! This would b great to know!!

mzbac commented 1 year ago

If you want to host the API in Colab, you may have to use something like ngrok to do the tunneling. The main script for running the API will mostly be the same.

ashercn97 commented 1 year ago

Okay i will look into it! Ty!

ashercn97 commented 1 year ago

What part would i change to use ngrok?

mzbac commented 1 year ago

Please refer to the ngrok documentation for tunneling -> https://ngrok.com/docs/getting-started/