run python3 test_httpserver_llava.py get ValueError: 64002 is not in list

[Aurorana]

run python3 test_httpserver_llava.py offset = input_ids.index(self.config.image_token_index) ValueError: 64002 is not in list

def test_streaming(args): url = f"{args.host}:{args.port}" response = requests.post( url + "/generate", json={ 'text' : 'A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human\'s questions. USER: description the video indetail \n Assistant:',

"text": "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: Describe this picture <|im_start|> <|im_end|>\n ASSISTANT:",

        "image_data": "examples/image1.webp",
        "sampling_params": {
            "temperature": 0,
            "max_new_tokens": 128,
        },
        "stream": True,
    },
    stream=True,
)
print(response)
prev = 0
for chunk in response.iter_lines(decode_unicode=False):
    chunk = chunk.decode("utf-8")
    if chunk and chunk.startswith("data:"):
        if chunk == "data: [DONE]":
            break
        data = json.loads(chunk[5:].strip("\n"))
        output = data["text"].strip()
        print(output[prev:], end="", flush=True)
        prev = len(output)
print("--------")

[apt-team-018]

facing same issues

[apt-team-018]

@Aurorana you have to use vision enabled toknizer, as we added extra token in vocab.

for llama3next you can use - lmms-lab/llama3-llava-next-8b-tokenizer

you just have to replace - preprocessor_config.json, special_tokens_map.json, tokenizer.json, tokenizer_config.json files

do not replace config.json

[github-actions[bot]]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.