runpod-workers / worker-vllm

The RunPod worker template for serving our large language model endpoints. Powered by vLLM.
MIT License
220 stars 85 forks source link

test_input.json is required #31

Closed wwydmanski closed 7 months ago

wwydmanski commented 8 months ago

When I try to run the worker, I'm getting the following error: WARN | test_input.json not found, exiting.

which prevents me from using the runpod worker.

alpayariyak commented 8 months ago

Hi @wwydmanski, could you please share more details? Does this still persist?

ashleykleynhans commented 8 months ago

This is normal when you run it locally, you can start the local API as follows:

python3 -u handler.py --rp_serve_api --rp_api_port 8000 --rp_api_host 0.0.0.0

Otherwise it defaults to looking for test_input.json instead.