Event loop stopped before Future completed -generator.generate_with_langchain_docs

[jackqdldd]

ragas 0.2.3 langchain 0.2.16 langchain-chroma 0.1.4 langchain-community 0.2.16 langchain-core 0.2.40 langchain-huggingface 0.1.0 langchain-openai 0.1.23 langchain-text-splitters 0.2.4 langchain-unstructured 0.1.4

File "/home/alg/qdl/evalscope/package/evalscope/backend/rag_eval/backend_manager.py", line 54, in run_ragas generate_testset(TestsetGenerationArguments(**testset_args)) File "/home/alg/qdl/evalscope/package/evalscope/backend/rag_eval/ragas/tasks/testset_generation.py", line 167, in generate_testset testset = generator.generate_with_langchain_docs( File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/testset/synthesizers/generate.py", line 129, in generate_with_langchain_docs return self.generate( File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/testset/synthesizers/generate.py", line 254, in generate raise e File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/testset/synthesizers/generate.py", line 251, in generate scenario_sample_list: t.List[t.List[BaseScenario]] = exec.results() File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 146, in results results = asyncio.run(_aresults()) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 141, in _aresults r = await future File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one return f.result() # May raise f.exception(). File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 36, in sema_coro return await coro File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 81, in wrapped_callable_async raise e File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 78, in wrapped_callable_async result = await callable(*args, **kwargs) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/testset/synthesizers/base.py", line 89, in generate_scenarios scenarios = await self._generate_scenarios( File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/testset/synthesizers/abstract_query.py", line 263, in _generate_scenarios common_concepts: t.List[Concepts] = run_async_batch( File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 167, in run_async_batch return executor.results() File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 146, in results results = asyncio.run(_aresults()) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/nest_asyncio.py", line 30, in run return loop.run_until_complete(task) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/nest_asyncio.py", line 98, in run_until_complete return f.result() File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/futures.py", line 201, in result raise self._exception.with_traceback(self._exception_tb) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/tasks.py", line 232, in __step result = coro.send(None) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 141, in _aresults r = await future File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one return f.result() # May raise f.exception(). File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/futures.py", line 201, in result raise self._exception.with_traceback(self._exception_tb) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/tasks.py", line 232, in __step result = coro.send(None) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 36, in sema_coro return await coro File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 81, in wrapped_callable_async raise e File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 78, in wrapped_callable_async result = await callable(*args, **kwargs) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/prompt/pydantic_prompt.py", line 130, in generate output_single = await self.generate_multiple( File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/prompt/pydantic_prompt.py", line 190, in generate_multiple resp = await llm.generate( File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/llms/base.py", line 100, in generate result = await agenerate_text_with_retry( File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped return await copy(fn, *args, **kwargs) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 111, in __call__ do = await self.iter(retry_state=retry_state) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 153, in iter result = await action(retry_state) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/tenacity/_utils.py", line 99, in inner return call(*args, **kwargs) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/tenacity/__init__.py", line 398, in <lambda> self._add_action_func(lambda rs: rs.outcome.result()) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.__get_result() File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 114, in __call__ result = await fn(*args, **kwargs) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/llms/base.py", line 220, in agenerate_text return await self.langchain_llm.agenerate_prompt( File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 787, in agenerate_prompt return await self.agenerate( File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 747, in agenerate raise exceptions[0] File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/asyncio/tasks.py", line 232, in __step result = coro.send(None) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 923, in _agenerate_with_cache result = await self._agenerate( File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/langchain_openai/chat_models/base.py", line 815, in _agenerate response = await self.async_client.create(**payload) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/openai/resources/chat/completions.py", line 1490, in create return await self._post( File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/openai/_base_client.py", line 1841, in post return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/openai/_base_client.py", line 1533, in request return await self._request( File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/openai/_base_client.py", line 1608, in _request raise APIConnectionError(request=request) from err openai.APIConnectionError: Connection error. 2024-10-31 17:28:08,770 - asyncio - ERROR - Task exception was never retrieved future: <Task finished name='Task-334' coro=<as_completed.<locals>.sema_coro() done, defined at /home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py:34> exception=RuntimeError('Event loop stopped before Future completed.')> Traceback (most recent call last): File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 36, in sema_coro return await coro File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 81, in wrapped_callable_async raise e File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 78, in wrapped_callable_async result = await callable(*args, **kwargs) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/testset/synthesizers/base.py", line 89, in generate_scenarios scenarios = await self._generate_scenarios( File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/testset/synthesizers/abstract_query.py", line 119, in _generate_scenarios themes: t.List[Themes] = run_async_batch( File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 167, in run_async_batch return executor.results() File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/ragas/executor.py", line 146, in results results = asyncio.run(_aresults()) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/nest_asyncio.py", line 30, in run return loop.run_until_complete(task) File "/home/alg/anaconda3/envs/evalscope/lib/python3.10/site-packages/nest_asyncio.py", line 96, in run_until_complete raise RuntimeError( RuntimeError: Event loop stopped before Future completed.

[jackqdldd]

The generator llm is remote api model,and the embeddings is local model

[jjmachan]

@jackqdldd is it a huggingface model?

[jackqdldd]

@jackqdldd is it a huggingface model?

no,the llm is Qwen2.5-72B-Instruct-AWQ from modelscope

[jackqdldd]

it seems the [Generating Scenarios:] was interrupted