Potential error in shutdown if manually cancelled

[stewartugelow]

I don't know if this is a bug or not. Just wanted to flag it for you.

I deleted the Microsoft Phi models to reinstall them as you suggested in the EOS discussion, and that threw a model not found error with the server because I hadn't restarted. I did control + c to stop it and got an error on shutdown that I hadn't seen before. There was quite a delay before it actually shut down.

^CINFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [7603]
INFO:     127.0.0.1:61200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO:     Shutting down
INFO:     Finished server process [7602]
ERROR:    Traceback (most recent call last):
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 741, in lifespan
    await receive()
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/uvicorn/lifespan/on.py", line 137, in receive
    return await self.receive_queue.get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/asyncio/queues.py", line 158, in get
    await getter
asyncio.exceptions.CancelledError

INFO:     Stopping parent process [7600]

[Blaizzy]

There are a request to the completions API during shut down. I think that's what caused the delay plus you deleted the model.

This seems to be edge case...

I will close for now, we can re-open it if it occurs more often.

[Blaizzy]

Btw, thank you very much for flagging these issues!

I appreciate it a lot :)

[stewartugelow]

My devs always joked that finding edge cases was my super power. :)

[Blaizzy]

I'm loving them!

Please keep 'em coming 🚀

[Blaizzy]

Indeed, it is your super power.

Could you share your Twitter I want to tag on a post about your amazing work here

[stewartugelow]

Your project came along at the perfect time for me and will have a big impact on my work. I think it's amazing, and It's a genuine pleasure to help out. I'm @ugelow on Twitter.

PS Happy to look at documentation when you have some ready!