Fix bug where we attempt to close connection when there is none

[advay-modal]

Example of error happening in the wild: https://modal.com/jina-ai/prod/logs/ap-4J2oDxFbGtJqR8jrFhpuEJ?functionId=fu-G37JXo7zaXTanMdvwmyRHE&inputId=in-01J1E0AWMA28TSQNT5WFQZWARP