Evaluating LlamaIndex example hangs

[yjlee1011]

I am new to ragas and trying to learn how to evaluate LlamaIndex. When I tried the following code, which is copy-and-pasted from https://github.com/explodinggradients/ragas/blob/main/docs/howtos/integrations/llamaindex.ipynb, the code just hangs forever after showing "evaluating with [faithfulness]" and "0%| | 0/1 [00:00<?, ?it/s]".

Executed codes: import nest_asyncio from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext

nest_asyncio.apply()

ocuments = SimpleDirectoryReader("./nyc_wikipedia/").load_data() vector_index = VectorStoreIndex.from_documents( documents, service_context=ServiceContext.from_defaults(chunk_size=512) )

query_engine = vector_index.as_query_engine()

eval_questions = [ "What is the population of New York City as of 2020?", "Which borough of New York City has the highest population?", "What is the economic significance of New York City?", "How did New York City get its name?", "What is the significance of the Statue of Liberty in New York City?", ]

eval_answers = [ "8,804,000", # incorrect answer "Queens", # incorrect answer "New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.", "New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.", "The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.", ]

eval_answers = [[a] for a in eval_answers]

from ragas.metrics import ( faithfulness, answer_relevancy, context_precision, context_recall, ) from ragas.metrics.critique import harmfulness

metrics = [ faithfulness, answer_relevancy, context_precision, context_recall, harmfulness, ]

from ragas.llama_index import evaluate

result = evaluate(query_engine, metrics, eval_questions, eval_answers)

Can anyone help me resolve the issue?

Thanks

[praveenck06]

can you share the statements that is printed before it hangs ?

[yjlee1011]

I am not sure what you are asking.

This is the last Python code I was executing: from ragas.llama_index import evaluate result = evaluate(query_engine, metrics, eval_questions, eval_answers)

This is the message I got: evaluating with [faithfulness] 0%| | 0/1 [03:31<?, ?it/s]

The progress bar does not move at all. So I had to stop the process with Control-C.

[jjmachan]

hey @yjlee1011 which version of ragas are you using?

[yjlee1011]

I have ragas (Version: 0.0.22) installed on my Mac.

[jjmachan]

that is strange but I would suggest you keep track of #245 now. We will release a new version of Ragas v0.1 today.

but in the meantime I don't have a fix for you sadly :(