search

jina-ai / dalle-flow

🌊 A Human-in-the-Loop workflow for creating HD images from text
grpcs://dalle-flow.dev.jina.ai
2.83k stars 211 forks source link

eventloop keeps crashing in the background #67

Closed yesbroc closed 2 years ago

yesbroc commented 2 years ago

Exception in thread Thread-22: Traceback (most recent call last): File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/usr/local/lib/python3.7/dist-packages/jina/helper.py", line 1292, in run self.result = asyncio.run(func(*args, kwargs)) File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run return loop.run_until_complete(main) File "/usr/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete return future.result() File "/usr/local/lib/python3.7/dist-packages/jina/clients/mixin.py", line 176, in _get_results async for resp in c._get_results(*args, kwargs): File "/usr/local/lib/python3.7/dist-packages/jina/clients/base/grpc.py", line 96, in _get_results logger=self.logger, File "/usr/local/lib/python3.7/dist-packages/jina/clients/helper.py", line 81, in callback_exec raise BadServer(response.header) jina.excepts.BadServer: request_id: "455de4daf4034b088f72b4816848e152" status { code: ERROR description: "BadClient(\'gRPC error: StatusCode.UNKNOWN Received http2 header with status: 524\')" exception { name: "BadClient" args: "gRPC error: StatusCode.UNKNOWN Received http2 header with status: 524" stacks: "Traceback (most recent call last):\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/jina/clients/base/grpc.py\", line 83, in _get_results\n async for resp in stub.Call(\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/grpc/aio/_call.py\", line 326, in _fetch_stream_responses\n await self._raise_for_status()\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/grpc/aio/_call.py\", line 236, in _raise_for_status\n raise _create_rpc_error(await self.initial_metadata(), await\n" stacks: "grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:\n\tstatus = StatusCode.UNKNOWN\n\tdetails = \"Received http2 header with status: 524\"\n\tdebug_error_string = \"{\"created\":\"@1657811503.084804400\",\"description\":\"Error received from peer ipv4:188.114.96.3:2096\",\"file\":\"src/core/lib/surface/call.cc\",\"file_line\":966,\"grpc_message\":\"Received http2 header with status: 524\",\"grpc_status\":2}\"\n>\n" stacks: "\nThe above exception was the direct cause of the following exception:\n\n" stacks: "Traceback (most recent call last):\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/jina/serve/runtimes/worker/init.py\", line 165, in process_data\n return await self._data_request_handler.handle(requests=requests)\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/jina/serve/runtimes/request_handlers/data_request_handler.py\", line 150, in handle\n return_data = await self._executor.acall(\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/jina/serve/executors/init.py\", line 288, in acall\n return await self.__acall_endpoint__(req_endpoint, kwargs)\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/jina/serve/executors/init.py\", line 309, in __acall_endpoint__\n return await func(self, *kwargs)\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/jina/serve/executors/decorators.py\", line 207, in arg_wrapper\n return await fn(executor_instance, args, kwargs)\n" stacks: " File \"/workdir/executor/executor.py\", line 69, in diffusion\n await self.run_glid3(d, d.text, skip_rate=skip_rate, num_images=num_images)\n" stacks: " File \"/workdir/executor/executor.py\", line 48, in run_glid3\n await do_run(args)\n" stacks: " File \"/workdir/glid-3-xl/dalle_flow_glid3/sample.py\", line 118, in do_run\n text_emb_clip_blank = await clip_c.aencode(\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/clip_client/client.py\", line 272, in aencode\n async for da in self._async_client.post(\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/jina/clients/mixin.py\", line 257, in post\n async for result in c._get_results(\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/jina/clients/base/grpc.py\", line 134, in _get_results\n raise BadClient(msg) from err\n" stacks: "jina.excepts.BadClient: gRPC error: StatusCode.UNKNOWN Received http2 header with status: 524\n" executor: "GLID3Diffusion" } } exec_endpoint: "/"

AttributeError Traceback (most recent call last) /usr/local/lib/python3.7/dist-packages/jina/helper.py in run_async(func, *args, **kwargs) 1306 try: -> 1307 return thread.result 1308 except AttributeError:

AttributeError: '_RunThread' object has no attribute 'result'

During handling of the above exception, another exception occurred:

BadClient Traceback (most recent call last) 7 frames

in time(self, line, cell, local_ns) in () [/usr/local/lib/python3.7/dist-packages/jina/helper.py](https://localhost:8080/#) in run_async(func, *args, **kwargs) 1310 1311 raise BadClient( -> 1312 'something wrong when running the eventloop, result can not be retrieved' 1313 ) 1314 else: BadClient: something wrong when running the eventloop, result can not be retrieved i dont know how to fix this
davvoz commented 2 years ago

Same here from Google Colab , when try to rerun from cell 3 to cell 2 `Exception in thread Thread-27: Traceback (most recent call last): File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/usr/local/lib/python3.7/dist-packages/jina/helper.py", line 1299, in run self.result = asyncio.run(func(*args, kwargs)) File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run return loop.run_until_complete(main) File "/usr/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete return future.result() File "/usr/local/lib/python3.7/dist-packages/jina/clients/mixin.py", line 256, in _get_results async for resp in c._get_results(*args, kwargs): File "/usr/local/lib/python3.7/dist-packages/jina/clients/base/grpc.py", line 100, in _get_results logger=self.logger, File "/usr/local/lib/python3.7/dist-packages/jina/clients/helper.py", line 81, in callback_exec raise BadServer(response.header) jina.excepts.BadServer: request_id: "400c0106da3d4b009d9d617cd788bc50" status { code: ERROR description: "TypeError(\"\'NoneType\' object is not subscriptable\")" exception { name: "TypeError" args: "\'NoneType\' object is not subscriptable" stacks: "Traceback (most recent call last):\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/jina/serve/runtimes/worker/init.py\", line 181, in process_data\n result = await self._data_request_handler.handle(requests=requests)\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/jina/serve/runtimes/request_handlers/data_request_handler.py\", line 152, in handle\n return_data = await self._executor.acall(\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/jina/serve/executors/init.py\", line 299, in acall\n return await self.__acall_endpoint__(req_endpoint, kwargs)\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/jina/serve/executors/init.py\", line 322, in __acall_endpoint__\n return func(self, *kwargs)\n" stacks: " File \"/usr/local/lib/python3.8/dist-packages/jina/serve/executors/decorators.py\", line 213, in arg_wrapper\n return fn(executor_instance, args, kwargs)\n" stacks: " File \"/workdir/executor/executor.py\", line 84, in diffusion\n self.run_glid3(d, d.text, skip_rate=skip_rate, num_images=num_images)\n" stacks: " File \"/workdir/executor/executor.py\", line 56, in run_glid3\n do_run(args, d.embedding, self.blank_bert_embedding, self.blank_clip_embedding)\n" stacks: " File \"/workdir/glid-3-xl/dalle_flow_glid3/sample.py\", line 19, in do_run\n text_emb_clip = np.repeat(text_emb_clip[np.newaxis, :], runtime_args.batch_size, axis=0)\n" stacks: "TypeError: \'NoneType\' object is not subscriptable\n" executor: "GLID3Diffusion" } } exec_endpoint: "/" target_executor: ""

AttributeError Traceback (most recent call last) /usr/local/lib/python3.7/dist-packages/jina/helper.py in run_async(func, *args, **kwargs) 1313 try: -> 1314 return thread.result 1315 except AttributeError:

AttributeError: '_RunThread' object has no attribute 'result'

During handling of the above exception, another exception occurred:

BadClient Traceback (most recent call last)

in 3 frames [/usr/local/lib/python3.7/dist-packages/docarray/document/mixins/sugar.py](https://localhost:8080/#) in post(self, *args, **kwargs) 90 91 _tmp = DocumentArray(self) ---> 92 return _tmp.post(*args, **kwargs)[0] [/usr/local/lib/python3.7/dist-packages/docarray/array/mixins/post.py](https://localhost:8080/#) in post(self, host, show_progress, batch_size, parameters, **kwargs) 113 request_size=batch_size, 114 parameters=parameters, --> 115 **kwargs, 116 ) 117 else: [/usr/local/lib/python3.7/dist-packages/jina/clients/mixin.py](https://localhost:8080/#) in post(self, on, inputs, on_done, on_error, on_always, parameters, target_executor, request_size, show_progress, continue_on_error, return_responses, **kwargs) 273 parameters=parameters, 274 request_size=request_size, --> 275 **kwargs, 276 ) 277 [/usr/local/lib/python3.7/dist-packages/jina/helper.py](https://localhost:8080/#) in run_async(func, *args, **kwargs) 1317 1318 raise BadClient( -> 1319 'something wrong when running the eventloop, result can not be retrieved' 1320 ) 1321 else: BadClient: something wrong when running the eventloop, result can not be retrieved`
delgermurun commented 2 years ago

@davvoz maybe https://github.com/jina-ai/dalle-flow/issues/100#issuecomment-1237837986 this will help.

davvoz commented 2 years ago

@delgermurun Thank you ! So i modify step 3 adding fav.embedding = doc.embedding

 dfav_id = 2
fav = diffused[dfav_id]
fav.embedding = doc.embedding
fav.display()