search

runpod-workers / worker-vllm

The RunPod worker template for serving our large language model endpoints. Powered by vLLM.
MIT License
220 stars 85 forks source link

PSA: Things break if you don't set TOKENIZER variable #14

Closed Palmik closed 8 months ago

Palmik commented 10 months ago

For some reason, when you don't specify it, it results in tokenizer being '' rather than None. Adding the following snippet fixes it:

if len(TOKENIZER) == 0:
    TOKENIZER = None
ashleykleynhans commented 9 months ago

@Palmik nice find, would also be nice to create a PR to fix it though 😬 . I've created one though.

ricardo-st commented 9 months ago

Good find, was going crazy about this. Would be even better to check for the following:

if not TOKENIZER:
    TOKENIZER = None

As this handles the case where TOKENIZER is already set to None, 0, or any other non standard way.

ashleykleynhans commented 9 months ago

Good find, was going crazy about this. Would be even better to check for the following:

if not TOKENIZER:
    TOKENIZER = None

As this handles the case where TOKENIZER is already set to None, 0, or any other non standard way.

My PR does exactly this, would be nice if it could get merged: https://github.com/runpod-workers/worker-vllm/pull/17/files