System Info

https://colab.research.google.com/github/oobabooga/text-generation-webui/blob/main/Colab-TextGen-GPU.ipynb

Once you choose to get the API it gives the error that it could not start cloudflared.

Here's the code of the google collab :

@title 2. Launch the web UI

@markdown If unsure about the branch, write "main" or leave it blank.

import torch from pathlib import Path

if Path.cwd().name != 'text-generation-webui': print("Installing the webui...")

!git clone https://github.com/oobabooga/text-generation-webui %cd text-generation-webui

torver = torch.version print(f"TORCH: {torver}") is_cuda118 = '+cu118' in torver # 2.1.0+cu118 is_cuda117 = '+cu117' in torver # 2.0.1+cu117

textgen_requirements = open('requirements.txt').read().splitlines() if is_cuda117: textgen_requirements = [req.replace('+cu121', '+cu117').replace('+cu122', '+cu117').replace('torch2.1', 'torch2.0') for req in textgen_requirements] elif is_cuda118: textgen_requirements = [req.replace('+cu121', '+cu118').replace('+cu122', '+cu118') for req in textgen_requirements] with open('temp_requirements.txt', 'w') as file: file.write('\n'.join(textgen_requirements))

!pip install -r extensions/api/requirements.txt --upgrade !pip install -r temp_requirements.txt --upgrade

print("\033[1;32;1m\n --> If you see a warning about \"previously imported packages\", just ignore it.\033[0;37;0m") print("\033[1;32;1m\n --> There is no need to restart the runtime.\n\033[0;37;0m")

try: import flash_attn except: !pip uninstall -y flash_attn

Parameters

model_url = "https://huggingface.co/TheBloke/HornyEchidna-13B-v0.1-GPTQ" #@param {type:"string"} branch = "main" #@param {type:"string"} command_line_flags = "--n-gpu-layers 128 --load-in-4bit --use_double_quant" #@param {type:"string"} api = True #@param {type:"boolean"}

if api: for param in ['--api', '--public-api']: if param not in command_line_flags: command_line_flags += f" {param}"

model_url = model_url.strip() if model_url != "": if not model_url.startswith('http'): model_url = 'https://huggingface.co/' + model_url

# Download the model
url_parts = model_url.strip('/').strip().split('/')
output_folder = f"{url_parts[-2]}_{url_parts[-1]}"
branch = branch.strip('"\' ')
if branch.strip() != '':
    output_folder += f"_{branch}"
    !python download-model.py {model_url} --branch {branch}
else:
    !python download-model.py {model_url}

else: output_folder = ""

Start the web UI

cmd = f"python server.py --share" if output_folder != "": cmd += f" --model {output_folder}" cmd += f" {command_line_flags}" print(cmd) !$cmd

Information

[X] Docker
[ ] The CLI directly

Tasks

[X] An officially supported command
[ ] My own modifications

Reproduction

Traceback (most recent call last): File "/content/text-generation-webui/extensions/api/util.py", line 112, in _start_cloudflared public_url = _run_cloudflared(port, port + 1) File "/usr/local/lib/python3.10/dist-packages/flask_cloudflared.py", line 137, in _run_cloudflared raise Exception(f"! Can't connect to Cloudflare Edge") Exception: ! Can't connect to Cloudflare Edge Exception in thread Thread-2 (_start_cloudflared): Traceback (most recent call last): File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/content/text-generation-webui/extensions/api/util.py", line 122, in _start_cloudflared raise Exception('Could not start cloudflared.') Exception: Could not start cloudflared.

Expected behavior

It should gives 2 link like : https://mortgage-cabinets-boy-bbc.trycloudflare.com/api

                 wss://insulin-regional-customized-courts.trycloudflare.com/api/v1/stream

huggingface / text-generation-inference

Google Collab TextGen-GPU API doesn't work anymore. #1229