Closed FelixTJDietrich closed 1 month ago
The Webhook Ingest service dropped a message due to Internal Server Error.
2024-08-21T12:26:22.558721383Z nats.errors.NoRespondersError: nats: no responders available for request
2024-08-21T12:26:22.558863934Z nats.js.errors.NoStreamResponseError: nats: no response from stream
2024-08-20T17:48:21.034180519Z nats.errors.TimeoutError: nats: timeout
Buffer/queue payloads and retry once NATS is responsive again.
2024-08-21T12:26:21.877466355Z INFO: 172.23.0.2:40320 - "POST /github HTTP/1.1" 500 Internal Server Error 2024-08-21T12:26:21.878463292Z ERROR: Exception in ASGI application 2024-08-21T12:26:21.878491042Z Traceback (most recent call last): 2024-08-21T12:26:21.878494432Z File "/usr/local/lib/python3.12/site-packages/nats/js/client.py", line 181, in publish 2024-08-21T12:26:21.878497792Z msg = await self._nc.request( 2024-08-21T12:26:21.878500622Z ^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-21T12:26:21.878503043Z File "/usr/local/lib/python3.12/site-packages/nats/aio/client.py", line 1026, in request 2024-08-21T12:26:21.878505603Z raise errors.NoRespondersError 2024-08-21T12:26:21.878508043Z nats.errors.NoRespondersError: nats: no responders available for request 2024-08-21T12:26:21.878510542Z 2024-08-21T12:26:21.878513213Z During handling of the above exception, another exception occurred: 2024-08-21T12:26:21.878515773Z 2024-08-21T12:26:21.878534983Z Traceback (most recent call last): 2024-08-21T12:26:21.878537552Z File "/usr/local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi 2024-08-21T12:26:21.878540112Z result = await app( # type: ignore[func-returns-value] 2024-08-21T12:26:21.878542533Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-21T12:26:21.878544933Z File "/usr/local/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__ 2024-08-21T12:26:21.878547753Z return await self.app(scope, receive, send) 2024-08-21T12:26:21.878550213Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-21T12:26:21.878552603Z File "/usr/local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__ 2024-08-21T12:26:21.878555283Z await super().__call__(scope, receive, send) 2024-08-21T12:26:21.878557713Z File "/usr/local/lib/python3.12/site-packages/starlette/applications.py", line 123, in __call__ 2024-08-21T12:26:21.878560213Z await self.middleware_stack(scope, receive, send) 2024-08-21T12:26:21.878562643Z File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__ 2024-08-21T12:26:21.878565173Z raise exc 2024-08-21T12:26:21.878567573Z File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__ 2024-08-21T12:26:21.878570093Z await self.app(scope, receive, _send) 2024-08-21T12:26:21.878572493Z File "/usr/local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 65, in __call__ 2024-08-21T12:26:21.878575093Z await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) 2024-08-21T12:26:21.878577573Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app 2024-08-21T12:26:21.878580093Z raise exc 2024-08-21T12:26:21.878582453Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app 2024-08-21T12:26:21.878584943Z await app(scope, receive, sender) 2024-08-21T12:26:21.878587913Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 754, in __call__ 2024-08-21T12:26:21.878590503Z await self.middleware_stack(scope, receive, send) 2024-08-21T12:26:21.878593413Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 774, in app 2024-08-21T12:26:21.878596563Z await route.handle(scope, receive, send) 2024-08-21T12:26:21.878599083Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 295, in handle 2024-08-21T12:26:21.878601593Z await self.app(scope, receive, send) 2024-08-21T12:26:21.878604003Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 77, in app 2024-08-21T12:26:21.878606463Z await wrap_app_handling_exceptions(app, request)(scope, receive, send) 2024-08-21T12:26:21.878612323Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app 2024-08-21T12:26:21.878615263Z raise exc 2024-08-21T12:26:21.878617644Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app 2024-08-21T12:26:21.878620433Z await app(scope, receive, sender) 2024-08-21T12:26:21.878622833Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 74, in app 2024-08-21T12:26:21.878625293Z response = await f(request) 2024-08-21T12:26:21.878627703Z ^^^^^^^^^^^^^^^^ 2024-08-21T12:26:21.878630084Z File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 278, in app 2024-08-21T12:26:21.878632564Z raw_response = await run_endpoint_function( 2024-08-21T12:26:21.878634983Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-21T12:26:21.878637484Z File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 191, in run_endpoint_function 2024-08-21T12:26:21.878640113Z return await dependant.call(**values) 2024-08-21T12:26:21.878642534Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-21T12:26:21.878644924Z File "/code/app/main.py", line 66, in github_webhook 2024-08-21T12:26:21.878647384Z await nats_client.js.publish(subject, body) 2024-08-21T12:26:21.878649784Z File "/usr/local/lib/python3.12/site-packages/nats/js/client.py", line 188, in publish 2024-08-21T12:26:21.878652273Z raise nats.js.errors.NoStreamResponseError 2024-08-21T12:26:21.878654694Z nats.js.errors.NoStreamResponseError: nats: no response from stream 2024-08-21T12:26:22.557690946Z INFO: 172.23.0.2:40336 - "POST /github HTTP/1.1" 500 Internal Server Error 2024-08-21T12:26:22.558699804Z ERROR: Exception in ASGI application 2024-08-21T12:26:22.558705503Z Traceback (most recent call last): 2024-08-21T12:26:22.558708354Z File "/usr/local/lib/python3.12/site-packages/nats/js/client.py", line 181, in publish 2024-08-21T12:26:22.558711524Z msg = await self._nc.request( 2024-08-21T12:26:22.558713954Z ^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-21T12:26:22.558716414Z File "/usr/local/lib/python3.12/site-packages/nats/aio/client.py", line 1026, in request 2024-08-21T12:26:22.558718954Z raise errors.NoRespondersError 2024-08-21T12:26:22.558721383Z nats.errors.NoRespondersError: nats: no responders available for request 2024-08-21T12:26:22.558723984Z 2024-08-21T12:26:22.558726454Z During handling of the above exception, another exception occurred: 2024-08-21T12:26:22.558728914Z 2024-08-21T12:26:22.558731284Z Traceback (most recent call last): 2024-08-21T12:26:22.558733694Z File "/usr/local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi 2024-08-21T12:26:22.558751643Z result = await app( # type: ignore[func-returns-value] 2024-08-21T12:26:22.558754213Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-21T12:26:22.558756633Z File "/usr/local/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__ 2024-08-21T12:26:22.558759153Z return await self.app(scope, receive, send) 2024-08-21T12:26:22.558761633Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-21T12:26:22.558764113Z File "/usr/local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__ 2024-08-21T12:26:22.558766693Z await super().__call__(scope, receive, send) 2024-08-21T12:26:22.558769103Z File "/usr/local/lib/python3.12/site-packages/starlette/applications.py", line 123, in __call__ 2024-08-21T12:26:22.558771583Z await self.middleware_stack(scope, receive, send) 2024-08-21T12:26:22.558773993Z File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__ 2024-08-21T12:26:22.558776483Z raise exc 2024-08-21T12:26:22.558778893Z File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__ 2024-08-21T12:26:22.558781613Z await self.app(scope, receive, _send) 2024-08-21T12:26:22.558784113Z File "/usr/local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 65, in __call__ 2024-08-21T12:26:22.558786603Z await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) 2024-08-21T12:26:22.558789063Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app 2024-08-21T12:26:22.558791563Z raise exc 2024-08-21T12:26:22.558793913Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app 2024-08-21T12:26:22.558796383Z await app(scope, receive, sender) 2024-08-21T12:26:22.558799023Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 754, in __call__ 2024-08-21T12:26:22.558801554Z await self.middleware_stack(scope, receive, send) 2024-08-21T12:26:22.558803973Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 774, in app 2024-08-21T12:26:22.558806493Z await route.handle(scope, receive, send) 2024-08-21T12:26:22.558808914Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 295, in handle 2024-08-21T12:26:22.558811404Z await self.app(scope, receive, send) 2024-08-21T12:26:22.558813794Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 77, in app 2024-08-21T12:26:22.558816253Z await wrap_app_handling_exceptions(app, request)(scope, receive, send) 2024-08-21T12:26:22.558818733Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app 2024-08-21T12:26:22.558821213Z raise exc 2024-08-21T12:26:22.558823554Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app 2024-08-21T12:26:22.558829314Z await app(scope, receive, sender) 2024-08-21T12:26:22.558831754Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 74, in app 2024-08-21T12:26:22.558834244Z response = await f(request) 2024-08-21T12:26:22.558836654Z ^^^^^^^^^^^^^^^^ 2024-08-21T12:26:22.558839054Z File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 278, in app 2024-08-21T12:26:22.558841564Z raw_response = await run_endpoint_function( 2024-08-21T12:26:22.558843974Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-21T12:26:22.558846404Z File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 191, in run_endpoint_function 2024-08-21T12:26:22.558848934Z return await dependant.call(**values) 2024-08-21T12:26:22.558851694Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-21T12:26:22.558854124Z File "/code/app/main.py", line 66, in github_webhook 2024-08-21T12:26:22.558856554Z await nats_client.js.publish(subject, body) 2024-08-21T12:26:22.558858964Z File "/usr/local/lib/python3.12/site-packages/nats/js/client.py", line 188, in publish 2024-08-21T12:26:22.558861504Z raise nats.js.errors.NoStreamResponseError 2024-08-21T12:26:22.558863934Z nats.js.errors.NoStreamResponseError: nats: no response from stream
2024-08-20T17:48:20.985777907Z INFO: 172.23.0.2:41276 - "POST /github HTTP/1.1" 500 Internal Server Error 2024-08-20T17:48:21.032638699Z ERROR: Exception in ASGI application 2024-08-20T17:48:21.032657799Z Traceback (most recent call last): 2024-08-20T17:48:21.032661369Z File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for 2024-08-20T17:48:21.032664299Z return await fut 2024-08-20T17:48:21.032666800Z ^^^^^^^^^ 2024-08-20T17:48:21.032671960Z 2024-08-20T17:48:21.032674469Z The above exception was the direct cause of the following exception: 2024-08-20T17:48:21.032677130Z 2024-08-20T17:48:21.032679570Z Traceback (most recent call last): 2024-08-20T17:48:21.032693280Z File "/usr/local/lib/python3.12/site-packages/nats/aio/client.py", line 1061, in _request_new_style 2024-08-20T17:48:21.032696000Z return await asyncio.wait_for(future, timeout) 2024-08-20T17:48:21.032698450Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.032701170Z File "/usr/local/lib/python3.12/asyncio/tasks.py", line 519, in wait_for 2024-08-20T17:48:21.032703690Z async with timeouts.timeout(timeout): 2024-08-20T17:48:21.032706140Z File "/usr/local/lib/python3.12/asyncio/timeouts.py", line 115, in __aexit__ 2024-08-20T17:48:21.032708640Z raise TimeoutError from exc_val 2024-08-20T17:48:21.032711290Z TimeoutError 2024-08-20T17:48:21.032713740Z 2024-08-20T17:48:21.032716140Z During handling of the above exception, another exception occurred: 2024-08-20T17:48:21.032718630Z 2024-08-20T17:48:21.032721020Z Traceback (most recent call last): 2024-08-20T17:48:21.032723460Z File "/usr/local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi 2024-08-20T17:48:21.032726020Z result = await app( # type: ignore[func-returns-value] 2024-08-20T17:48:21.032728570Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.032731210Z File "/usr/local/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__ 2024-08-20T17:48:21.032733790Z return await self.app(scope, receive, send) 2024-08-20T17:48:21.032736450Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.032739230Z File "/usr/local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__ 2024-08-20T17:48:21.032741800Z await super().__call__(scope, receive, send) 2024-08-20T17:48:21.032744280Z File "/usr/local/lib/python3.12/site-packages/starlette/applications.py", line 123, in __call__ 2024-08-20T17:48:21.032746840Z await self.middleware_stack(scope, receive, send) 2024-08-20T17:48:21.032749310Z File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__ 2024-08-20T17:48:21.032751900Z raise exc 2024-08-20T17:48:21.032754490Z File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__ 2024-08-20T17:48:21.032757030Z await self.app(scope, receive, _send) 2024-08-20T17:48:21.032759480Z File "/usr/local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 65, in __call__ 2024-08-20T17:48:21.032762620Z await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) 2024-08-20T17:48:21.032765380Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app 2024-08-20T17:48:21.032767960Z raise exc 2024-08-20T17:48:21.032770400Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app 2024-08-20T17:48:21.032776281Z await app(scope, receive, sender) 2024-08-20T17:48:21.032781070Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 754, in __call__ 2024-08-20T17:48:21.032783620Z await self.middleware_stack(scope, receive, send) 2024-08-20T17:48:21.032786100Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 774, in app 2024-08-20T17:48:21.032788620Z await route.handle(scope, receive, send) 2024-08-20T17:48:21.032791100Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 295, in handle 2024-08-20T17:48:21.032793641Z await self.app(scope, receive, send) 2024-08-20T17:48:21.032796201Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 77, in app 2024-08-20T17:48:21.032798780Z await wrap_app_handling_exceptions(app, request)(scope, receive, send) 2024-08-20T17:48:21.032801300Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app 2024-08-20T17:48:21.032803860Z raise exc 2024-08-20T17:48:21.032806281Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app 2024-08-20T17:48:21.032808841Z await app(scope, receive, sender) 2024-08-20T17:48:21.032811310Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 74, in app 2024-08-20T17:48:21.032813841Z response = await f(request) 2024-08-20T17:48:21.032816310Z ^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.032818801Z File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 278, in app 2024-08-20T17:48:21.032821351Z raw_response = await run_endpoint_function( 2024-08-20T17:48:21.032823831Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.032826311Z File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 191, in run_endpoint_function 2024-08-20T17:48:21.032828881Z return await dependant.call(**values) 2024-08-20T17:48:21.032831341Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.032833771Z File "/code/app/main.py", line 66, in github_webhook 2024-08-20T17:48:21.032836281Z await nats_client.js.publish(subject, body) 2024-08-20T17:48:21.032838741Z File "/usr/local/lib/python3.12/site-packages/nats/js/client.py", line 181, in publish 2024-08-20T17:48:21.032841261Z msg = await self._nc.request( 2024-08-20T17:48:21.032843691Z ^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.032846131Z File "/usr/local/lib/python3.12/site-packages/nats/aio/client.py", line 1021, in request 2024-08-20T17:48:21.032848681Z msg = await self._request_new_style( 2024-08-20T17:48:21.032851371Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.032853871Z File "/usr/local/lib/python3.12/site-packages/nats/aio/client.py", line 1063, in _request_new_style 2024-08-20T17:48:21.032859621Z raise errors.TimeoutError 2024-08-20T17:48:21.032862131Z nats.errors.TimeoutError: nats: timeout 2024-08-20T17:48:21.033055882Z INFO: 172.23.0.2:41282 - "POST /github HTTP/1.1" 500 Internal Server Error 2024-08-20T17:48:21.033970387Z ERROR: Exception in ASGI application 2024-08-20T17:48:21.033975028Z Traceback (most recent call last): 2024-08-20T17:48:21.033977528Z File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for 2024-08-20T17:48:21.033980277Z return await fut 2024-08-20T17:48:21.033982808Z ^^^^^^^^^ 2024-08-20T17:48:21.033985288Z asyncio.exceptions.CancelledError 2024-08-20T17:48:21.033987908Z 2024-08-20T17:48:21.033991078Z The above exception was the direct cause of the following exception: 2024-08-20T17:48:21.033993598Z 2024-08-20T17:48:21.033996008Z Traceback (most recent call last): 2024-08-20T17:48:21.033998468Z File "/usr/local/lib/python3.12/site-packages/nats/aio/client.py", line 1061, in _request_new_style 2024-08-20T17:48:21.034001158Z return await asyncio.wait_for(future, timeout) 2024-08-20T17:48:21.034003598Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.034006068Z File "/usr/local/lib/python3.12/asyncio/tasks.py", line 519, in wait_for 2024-08-20T17:48:21.034008608Z async with timeouts.timeout(timeout): 2024-08-20T17:48:21.034011098Z File "/usr/local/lib/python3.12/asyncio/timeouts.py", line 115, in __aexit__ 2024-08-20T17:48:21.034013598Z raise TimeoutError from exc_val 2024-08-20T17:48:21.034016058Z TimeoutError 2024-08-20T17:48:21.034018468Z 2024-08-20T17:48:21.034020868Z During handling of the above exception, another exception occurred: 2024-08-20T17:48:21.034023348Z 2024-08-20T17:48:21.034025718Z Traceback (most recent call last): 2024-08-20T17:48:21.034028168Z File "/usr/local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 401, in run_asgi 2024-08-20T17:48:21.034030738Z result = await app( # type: ignore[func-returns-value] 2024-08-20T17:48:21.034033208Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.034035678Z File "/usr/local/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__ 2024-08-20T17:48:21.034038228Z return await self.app(scope, receive, send) 2024-08-20T17:48:21.034040648Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.034043078Z File "/usr/local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__ 2024-08-20T17:48:21.034045568Z await super().__call__(scope, receive, send) 2024-08-20T17:48:21.034048038Z File "/usr/local/lib/python3.12/site-packages/starlette/applications.py", line 123, in __call__ 2024-08-20T17:48:21.034054868Z await self.middleware_stack(scope, receive, send) 2024-08-20T17:48:21.034057368Z File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__ 2024-08-20T17:48:21.034059889Z raise exc 2024-08-20T17:48:21.034062308Z File "/usr/local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__ 2024-08-20T17:48:21.034064838Z await self.app(scope, receive, _send) 2024-08-20T17:48:21.034067278Z File "/usr/local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 65, in __call__ 2024-08-20T17:48:21.034070089Z await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) 2024-08-20T17:48:21.034072658Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app 2024-08-20T17:48:21.034075219Z raise exc 2024-08-20T17:48:21.034077628Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app 2024-08-20T17:48:21.034080188Z await app(scope, receive, sender) 2024-08-20T17:48:21.034082628Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 754, in __call__ 2024-08-20T17:48:21.034085179Z await self.middleware_stack(scope, receive, send) 2024-08-20T17:48:21.034087638Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 774, in app 2024-08-20T17:48:21.034090139Z await route.handle(scope, receive, send) 2024-08-20T17:48:21.034092638Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 295, in handle 2024-08-20T17:48:21.034095179Z await self.app(scope, receive, send) 2024-08-20T17:48:21.034097619Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 77, in app 2024-08-20T17:48:21.034100129Z await wrap_app_handling_exceptions(app, request)(scope, receive, send) 2024-08-20T17:48:21.034102619Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app 2024-08-20T17:48:21.034105149Z raise exc 2024-08-20T17:48:21.034107559Z File "/usr/local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app 2024-08-20T17:48:21.034110079Z await app(scope, receive, sender) 2024-08-20T17:48:21.034112529Z File "/usr/local/lib/python3.12/site-packages/starlette/routing.py", line 74, in app 2024-08-20T17:48:21.034115019Z response = await f(request) 2024-08-20T17:48:21.034133459Z ^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.034136979Z File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 278, in app 2024-08-20T17:48:21.034139489Z raw_response = await run_endpoint_function( 2024-08-20T17:48:21.034141919Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.034144329Z File "/usr/local/lib/python3.12/site-packages/fastapi/routing.py", line 191, in run_endpoint_function 2024-08-20T17:48:21.034150369Z return await dependant.call(**values) 2024-08-20T17:48:21.034152899Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.034155349Z File "/code/app/main.py", line 66, in github_webhook 2024-08-20T17:48:21.034157839Z await nats_client.js.publish(subject, body) 2024-08-20T17:48:21.034160289Z File "/usr/local/lib/python3.12/site-packages/nats/js/client.py", line 181, in publish 2024-08-20T17:48:21.034162829Z msg = await self._nc.request( 2024-08-20T17:48:21.034165259Z ^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.034167679Z File "/usr/local/lib/python3.12/site-packages/nats/aio/client.py", line 1021, in request 2024-08-20T17:48:21.034170199Z msg = await self._request_new_style( 2024-08-20T17:48:21.034172849Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-08-20T17:48:21.034175499Z File "/usr/local/lib/python3.12/site-packages/nats/aio/client.py", line 1063, in _request_new_style 2024-08-20T17:48:21.034178029Z raise errors.TimeoutError 2024-08-20T17:48:21.034180519Z nats.errors.TimeoutError: nats: timeout
Describe the bug
The Webhook Ingest service dropped a message due to Internal Server Error.
Proposed solution
Buffer/queue payloads and retry once NATS is responsive again.
Stacktrace
NoStreamResponse Error
TimeoutError