[Question]: LLama-Index v0.8.57 Much slower

[gich2009]

Question Validation

• [X] I have searched both the documentation and discord for an answer.

Question

What new retrieval approaches have been added to v0.8.57 compared to v0.8.56? The library is suddenly much slower than it was <0.8.57. Even for queries that do not require retrieval.

[dosubot[bot]]

🤖

Hi @gich2009! It's good to see you again. Hope you're doing well.

Give me a moment to look at your issue. I'll post a detailed answer shortly.

[gich2009]

Running in the same environment Conditions:

1. Using a chat engine
2. chat_mode = "context"
3. With 3 similar test queries

The average response time using v0.8.57 was 87s The average response time using v0.8.56 was 60s

[dosubot[bot]]

🤖

Based on the information provided, it seems that the slowdown you're experiencing with LlamaIndex v0.8.57 compared to v0.8.56 could be due to changes in the retrieval approaches in the AutoMergingRetriever class. Specifically, the _retrieve method in this class now includes a loop that continues to merge nodes until no more changes are made. This loop can potentially cause a slowdown, especially for large data sets, as it continues to merge nodes until no more changes can be made.

Here is the relevant code snippet from the AutoMergingRetriever class:

def _retrieve(self, query_bundle: QueryBundle) -> List[NodeWithScore]:
    """Retrieve nodes given query.

    Implemented by the user.
    """
    initial_nodes = self._vector_retriever.retrieve(query_bundle)

    cur_nodes, is_changed = self._try_merging(initial_nodes)
    while is_changed:
        cur_nodes, is_changed = self._try_merging(cur_nodes)

    # sort by similarity
    cur_nodes.sort(key=lambda x: x.get_score(), reverse=True)

    return cur_nodes

You can find this in the llama_index/retrievers/auto_merging_retriever.py file.

To resolve this issue, you could consider adding a limit to the number of iterations in the loop, or implement a more efficient merging algorithm. You could also consider adding a condition to skip the merging process for queries that do not require retrieval. This could potentially speed up the response time for these queries.

As for the specific changes made in the AutoMergingRetriever class from version v0.8.56 to v0.8.57, I wasn't able to find this information in the repository. It's possible that this information is available elsewhere, or I may have missed it. If you have more information about these changes, it could help in further diagnosing the issue.

Regarding similar issues, there was a Feature Request to improve the performance of the default vector index. However, it turned out that the issue was not with the default vector index but with the LLM response. The user was advised to check the logs correctly. This might not be directly related to your issue, but it could provide some insights.

I hope this information helps. If you have any more questions or need further clarification, feel free to ask.

Sources

• [Feature Request]: Improve performance of the default vector index
• llama_index/retrievers/auto_merging_retriever.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[gich2009]

Stream_chat() seems to work fine for v0.8.57 but the chat() method seems to be unreliable.

[gich2009]

There is a noticeable difference in retrieval for versions >0.8.57. Just tried 0.8.59 and the response is still slower than 0.8.56. The context keeping of the chat_engine also seems to have degraded. Can someone explain what major change has been made so that I can update my code base accordingly?

[logan-markewich]

@gich2009 there has been no major changes (that I know of). Are you using OpenAI? Maybe OpenAI updated their models (it does happen)

Can you re-produce directly if you install 0.8.56 vs 0.8.59? Would be happy to debug if you can provide a test case.

[logan-markewich]

The changelog is here. I don't see any concerning changes https://docs.llamaindex.ai/en/stable/changes/changelog.html

[gich2009]

Hi @logan-markewich, thanks for responding. I am using GPT-3.5-turbo. Let me run a test case and share the example output for both 0.8.56 and 0.8.59.

I don't think its an openai models issue because the test case results are poor for >0.8.57 but when I revert back to 0.8.56, the test case results are very good. My test cases are majorly written to evaluate speed and context.

[gich2009]

Here you go @logan-markewich

Test conditions:

1. GPT 3.5 turbo
2. Chat engine in context mode. All other factors are held constant and can be provided if needed.

Issue:

=v0.8.57 responses are very verbose therefore:

1. The speed is reduced
2. The context keeping is degraded.

Test cases: [ {"prompt": "Who is ACMI?"}, {"prompt": "What do they do?"}, {"prompt": "Cool, why is it important?"}, ]

Response for v0.8.56:

Response 1

ACMI refers to the Africa Carbon Markets Initiative. It is a coalition of organizations focused on climate impact, clean energy, and sustainable development. The initiative was launched by the Global Energy Alliance for People and Planet (GEAPP), Sustainable Energy for All (SEforALL), and the United Nations Economic Commission for Africa (UNECA), with support from the UN Climate Change High-Level Champions. ACMI aims to accelerate the growth of Africa's voluntary carbon markets and is led by a 13-person steering committee of influential African leaders and carbon market experts.

Response 2

The Africa Carbon Markets Initiative (ACMI) works towards accelerating the growth of voluntary carbon markets in Africa. Their main objectives include:

1. Promoting sustainable development: ACMI aims to support projects and initiatives that contribute to sustainable development in Africa, such as renewable energy projects, energy efficiency measures, and sustainable agriculture practices.

2. Facilitating access to carbon finance: ACMI seeks to enhance access to carbon finance for African businesses and organizations by providing guidance, capacity building, and technical assistance. They aim to help African entities navigate the complexities of carbon markets and develop projects that generate carbon credits.

3. Building capacity and knowledge: ACMI focuses on building the capacity of African stakeholders, including governments, businesses, and civil society organizations, to participate effectively in carbon markets. They provide training, workshops, and knowledge-sharing platforms to enhance understanding and expertise in carbon finance and project development.

4. Advocacy and policy support: ACMI advocates for supportive policies and regulations that enable the growth of carbon markets in Africa. They engage with policymakers, governments, and international organizations to promote favorable conditions for carbon market development and implementation.

Overall, ACMI plays a crucial role in promoting sustainable development, facilitating access to carbon finance, building capacity, and advocating for supportive policies to drive the growth of voluntary carbon markets in Africa.

Response 3

The Africa Carbon Markets Initiative (ACMI) is important for several reasons:

1. Climate change mitigation: ACMI focuses on promoting projects and initiatives that reduce greenhouse gas emissions. By facilitating access to carbon finance and supporting the development of sustainable projects, ACMI contributes to mitigating climate change and reducing the carbon footprint of African countries.

2. Sustainable development: ACMI's efforts align with the United Nations Sustainable Development Goals (SDGs). By promoting sustainable practices in sectors such as renewable energy, energy efficiency, and agriculture, ACMI helps African countries achieve sustainable development objectives, including poverty alleviation, access to clean energy, and environmental conservation.

3. Economic opportunities: Voluntary carbon markets provide economic opportunities for African businesses and organizations. By participating in carbon markets, African entities can generate revenue through the sale of carbon credits, which can be reinvested in sustainable projects and contribute to local economic growth.

4. Capacity building and knowledge transfer: ACMI's focus on building capacity and knowledge in carbon finance and project development helps African stakeholders gain expertise in these areas. This empowers them to effectively participate in carbon markets, attract investment, and implement sustainable projects, leading to long-term economic and environmental benefits.

5. International cooperation and collaboration: ACMI's work involves collaboration with international organizations, governments, and stakeholders. This fosters partnerships and knowledge exchange, enabling African countries to benefit from global best practices, technologies, and funding opportunities in the field of climate change mitigation and sustainable development.

Overall, ACMI's activities are important for addressing climate change, promoting sustainable development, creating economic opportunities, building capacity, and fostering international cooperation in Africa's transition to a low-carbon and sustainable future.

Time taken: 75.96979494199331 seconds

Response for v0.8.61(or any >=v0.8.57):

Response 1:

The Africa Carbon Markets Initiative (ACMI) is a coalition of organizations focused on climate impact, clean energy, and sustainable development. It aims to accelerate the growth of Africa's voluntary carbon markets. ACMI is led by a 13-person steering committee of influential African leaders and carbon market experts. The initiative aims to support the scaling of high integrity African carbon credits, create jobs, and support energy access, biodiversity, and livelihoods. ACMI is developing a roadmap of action programs to be implemented in the coming years, covering all aspects of the voluntary carbon market ecosystem. Integrity is a central focus for ACMI, and it will work with leading bodies to establish transparency and benefits-sharing standards. ACMI is not meant to compete with existing initiatives but rather to build on and reinforce ongoing efforts to scale voluntary carbon markets in Africa. The initiative intends to represent the broader voluntary carbon markets ecosystem and not individual organizations. ACMI's sponsors, partners, and steering committee members agree to disclose any potential direct benefits and recuse themselves from any efforts that could directly benefit themselves or their organizations. ACMI proposes action programs to scale project developers, develop national enabling ecosystems, build validation and verification capacity, create a more integrated African carbon market, reduce costs, leverage advanced market commitments, carbon neutral commodities, and advocacy to fuel demand for African credits. Cross-cutting action programs will pilot new project types and methodologies and explore diversified financing models for nature assets. Carbon credit integrity is a critical area of focus for ACMI, and it will work with leading bodies to establish transparency and benefits-sharing standards.

Response 2:

The role of carbon credits is to help reduce greenhouse gas emissions and combat climate change. Carbon credits are a form of tradeable permits that represent the right to emit one tonne of carbon dioxide or an equivalent greenhouse gas. They are used as a way to incentivize and finance projects that reduce emissions, such as renewable energy projects or energy efficiency initiatives.

Carbon credits are generated through offset projects, which are projects that reduce or remove greenhouse gas emissions. These projects are approved and certified by regulatory bodies or non-governmental organizations. The offset projects follow specific frameworks and protocols to ensure that they meet environmental standards and requirements.

Once an offset project is approved and certified, it can generate carbon credits. These credits can then be bought and sold on the carbon market. Companies or individuals that need to offset their own emissions can purchase these credits to compensate for their carbon footprint.

The carbon credit market is divided into compliance markets and voluntary markets. Compliance markets are regulated by international or governmental bodies and are used to meet mandatory emission reduction targets. Voluntary markets, on the other hand, are not regulated and are used by companies or individuals who want to voluntarily offset their emissions.

To ensure the integrity of carbon credits and the projects that generate them, there are various mechanisms in place. These include validation and verification requirements, which assess the eligibility and quantification of emission reductions or removals from offset projects. Registries are used to track the ownership and retirement of carbon credits to prevent double counting. Third-party auditors are also involved to ensure impartiality and limit conflicts of interest.

Overall, carbon credits play a crucial role in incentivizing and financing projects that reduce greenhouse gas emissions. They provide a market-based approach to addressing climate change and can help countries and companies meet their emission reduction targets.

Response 3:

Carbon credits are important for several reasons:

1. Climate Change Mitigation: Carbon credits help in reducing greenhouse gas emissions, which are the primary cause of climate change. By incentivizing and financing projects that reduce emissions, carbon credits contribute to mitigating climate change and its impacts.

2. Transition to a Low-Carbon Economy: Carbon credits play a crucial role in facilitating the transition to a low-carbon economy. By creating a market for emissions reductions, they encourage the development and deployment of renewable energy projects, energy efficiency initiatives, and other sustainable practices. This helps to shift away from fossil fuel-based industries and towards cleaner and more sustainable alternatives.

3. Encouraging Innovation and Investment: The availability of carbon credits provides financial incentives for businesses and organizations to invest in innovative technologies and practices that reduce emissions. This can spur technological advancements, research and development, and the adoption of sustainable practices across various sectors.

4. Supporting Sustainable Development: Many carbon offset projects, particularly in developing countries, have co-benefits beyond emissions reductions. These projects often contribute to sustainable development by providing access to clean energy, improving local livelihoods, protecting biodiversity, and promoting social and economic well-being in communities.

5. Corporate Social Responsibility: Carbon credits offer companies and organizations an opportunity to demonstrate their commitment to environmental sustainability and corporate social responsibility. By offsetting their own emissions, they can showcase their efforts to reduce their carbon footprint and contribute to global climate goals.

6. Compliance with Regulations and Targets: In compliance markets, carbon credits are essential for meeting mandatory emission reduction targets set by international agreements or governmental regulations. They provide a mechanism for companies and countries to achieve their emission reduction obligations and avoid penalties or sanctions.

7. Consumer Demand and Reputation: Increasingly, consumers are becoming more conscious of the environmental impact of the products and services they choose. Companies that demonstrate their commitment to reducing emissions and offsetting their carbon footprint through the use of carbon credits can enhance their reputation and attract environmentally conscious consumers.

Overall, carbon credits are important tools in the fight against climate change. They provide a market-based approach to reducing emissions, driving innovation, supporting sustainable development, and helping organizations meet their environmental goals.

Time taken: 113.71623640299367 seconds

As you can see, the bot loses context of the conversation because the first response is extremely wordy and gives way more information than is required by the prompt. This behaviour is consistent for tests done on any version greater than 0.8.56. Any versions lower than 0.8.56 maintain context very well.

[logan-markewich]

Hmmm. Are you able to share the setup code? Or even the test data? 🙏🏻

[gich2009]

Sure. Give me one minute to paste the relevant code and also send you another example of the weird behaviour before and after 0.8.56

[gich2009]

import openai import os from llama_index.llms import OpenAI from llama_index import ( ServiceContext, StorageContext, load_index_from_storage, set_global_service_context ) from llama_index.storage.docstore import SimpleDocumentStore from llama_index.storage.index_store import SimpleIndexStore from llama_index.vector_stores import SimpleVectorStore from llama_index.memory import ChatMemoryBuffer from llama_index.llms import ChatMessage, MessageRole from llama_index.indices.base import BaseIndex from dotenv import load_dotenv from llama_index.prompts import PromptTemplate from typing import Union import toml

if name=='main': import sys Current_dir = os.getcwd() Base_dir = os.path.dirname(Current_dir) sys.path.append(Base_dir) print(sys.path)

load_dotenv()

DEBUG = os.getenv('DEBUG') SECRET_KEY = os.getenv('SECRET_KEY') os.environ["OPENAI_API_KEY"] = SECRET_KEY openai.api_key = os.environ["OPENAI_API_KEY"]

SYSTEM_MESSAGE_RESOLUTION = 4 NO_OF_DOCS_TO_RETRIEVE = 3 CHAT_MODE = "context" # or react, context, best, condense-question, react, openai, simple DOMAIN = "CUSTOM" CONTEXT = "custom"

index: BaseIndex = None

if name == 'main': Base_dir = (os.path.dirname(os.getcwd())) else: Base_dir = (os.getcwd()) custom_path = os.path.join(Base_dir, "knowledge_base", f'{CONTEXT}')

Retrieve index from storage

storage_context = StorageContext.from_defaults( docstore=SimpleDocumentStore.from_persist_dir(persist_dir=custom_path), vector_store=SimpleVectorStore.from_persist_dir(persist_dir=custom_path), index_store=SimpleIndexStore.from_persist_dir(persist_dir=custom_path), )

index = load_index_from_storage(storage_context)

ctx = ServiceContext.from_defaults(llm=OpenAI(model="gpt-3.5-turbo", temperature=0)) set_global_service_context(ctx)

Previous and prior have different effects. Investigate and see which one to use and when.

TEMPLATE = """ -You are an AI assistant who helps people find information. -Do not speculate based on your previous knowledge. Answer STRICTLY and ONLY based on the documents provided. """

prompt_template = PromptTemplate(template=TEMPLATE)

system_message = ChatMessage( role=MessageRole.SYSTEM, content=prompt_template.format() )

memories = {} access_counts = {}

def customChatbot(input_text: str, chat_id: Union[int, str]): global index

access_counts[chat_id] = access_counts.get(chat_id, 0) + 1

if chat_id not in memories:
    memory = ChatMemoryBuffer(token_limit=1500)
    memory.put(system_message)
    memories[chat_id] = memory

memory = memories[chat_id]

# Check if access count for this chat_id is 4, if so, insert system message and reset count
if access_counts[chat_id] == SYSTEM_MESSAGE_RESOLUTION:
    memory.put(system_message)
    print(system_message)
    access_counts[chat_id] = 0  # Reset access count for this chat_id

chat_engine = index.as_chat_engine(
    chat_mode=CHAT_MODE, 
    similarity_top_k=NO_OF_DOCS_TO_RETRIEVE,
    memory=memory
)
response = chat_engine.chat(input_text)
memories[chat_id] = chat_engine._memory

return response.response

def custom(prompt: str, chat_id: Union[int, str]='1'): response = customChatbot(input_text=prompt, chat_id=chat_id) return response

if name == 'main': import time prompts = [

Comparison test with a query engine

        {"prompt": "Who is ACMI?"},
        {"prompt": "What do they do?"},
        {"prompt": "Cool, why is it important?"},

        # This test case is for observing context maintenance or history
        # Context test
        # {"prompt": "Hi, how are you?"},
        # {"prompt": "Who is ACMI?"},
        # {"prompt": "Why do they do this?"},
        # {"prompt": "Very interesting, tell me more about how I can contribute."},
        # {"prompt": "That sounds pretty boring. Could you make it more exciting?"},
        # {"prompt": "What is your role?"},
        # {"prompt": "Away from the digression, tell me more about the topic we were on"}

        # This test case is for observing the ability of the chat engine to retrieve information from different sources
        # Retrieval test
        # {"prompt": "Hi how are you?"},
        # {"prompt": "Tell me about the European Unions emission trading system"},
        # {"prompt": "Who is ACMI?"},
        # {"prompt": "Tell me about carbon credits and the water and sanitation sector"},
        # {"prompt": "Tell me about the types of carbon credits"},
        # {"prompt": "What was the first question I asked you?"},
        # {"prompt": "Why is it important to score carbon credits?"},
        # {"prompt": "Who is ACMI?"},
        # {"prompt": "Tell me about the carbon credit agreements between Singapore and Ghana."},
    ]
start_time = time.perf_counter()
for prompt in prompts:
    response = custom(prompt=prompt["prompt"])
    print(response)
end_time = time.perf_counter()
print(f"Time taken: {end_time - start_time} seconds\n")

Feel free to construct a vector store from a directory with 2 or more files and then load it, You can change the test cases to suit your data.

[gich2009]

It's almost as though it prioritizes synthesizing a response based on the retrieved nodes as opposed to answering the prompt. Look at these responses

Again, with similar test cases for 3 different agents that I constructed.

FOR >=v0.8.57: ''' { "id": 1, "time": "2023-11-06T02:19:18.411998", "prompt": "What is your role?", "response": "The Africa Climate Summit aims to showcase investment commitments in various sectors, including energy, and highlight tangible progress on ongoing initiatives and new partnerships. The summit calls for African Union Member States and supporting partners to champion the delivery of the Green Growth Agenda and Climate Finance plan, and present transformative opportunities for partnerships that can drive growth and attract new finance and investment. The summit also aims to provide a roadmap for the implementation of key declarations and announcements. The concept note emphasizes the need for substantial financing flows to support green growth plans in Africa and calls for a collective ask on specific financing needs. The summit seeks to rally African Heads of State and global financial sector leaders around tailored financial investment instruments that support growth opportunities on the continent. The expected outputs of the summit include a Nairobi Declaration on Green Growth and Climate Finance, a pathway to a new global financial deal, and commitments to specific initiatives such as energy transitions and carbon credit regulatory frameworks.", } '''

''' { "id": 1, "time": "2023-11-06T02:19:18.421086", "prompt": "What is your role?", "response": "In the context of the Voluntary Ecological Markets (VEMs), carbon credits play a crucial role in incentivizing and rewarding organizations or individuals for reducing or avoiding greenhouse gas (GHG) emissions. Carbon credits are a form of currency that represent a reduction or removal of CO2e (carbon dioxide equivalent) emissions from the atmosphere.\n\nThe role of carbon credits in VEMs is to provide a mechanism for organizations or individuals, known as project suppliers/sellers, to generate and trade these credits. Project suppliers/sellers are responsible for implementing or performing GHG reduction or avoidance activities, or owning the assets used in these activities. For example, a farmer who owns or leases land and performs a mitigation activity can be a project supplier/seller.\n\nTo create carbon credits, project suppliers/sellers need to generate evidence through a process that generates source data. This evidence is then verified against an independent \"Quality Standard\" by a standards body. The standards body is an organization that administers and governs the Quality Standards used by project suppliers/sellers. These standards establish the science-based definitions, metrics, and criteria upon which CO2e reductions can be credibly achieved by different types of projects and activities.\n\nOnce the project supplier/seller has demonstrated, validated, and verified reductions in accordance with the Quality Standards, the standards body issues verified credits for the ecological project. These credits can then be traded in the marketplace.\n\nIn summary, carbon credits play a vital role in VEMs by providing a means for project suppliers/sellers to generate and trade credits for their GHG reduction or avoidance activities. These credits are verified by a standards body, ensuring their credibility and integrity in the marketplace.", } '''

''' { "id": 1, "time": "2023-11-06T02:19:18.421086", "prompt": "What is your role?", "response": "As an AI representative of Verst Carbon, my role is to provide information and assistance regarding carbon markets and carbon credits. I can help answer any questions you may have about the role of carbon credits in achieving sustainability goals or any other related topics. How can I assist you today?", } '''

FOR <=v0.8.56: ''' { "id": 1, "time": "2023-11-06T02:44:59.596009", "prompt": "What is your role?", "response": "As an AI representative of the Africa Carbon Markets Initiative (ACMI), my role is to provide information and assistance regarding carbon markets and related topics. I can help answer questions, provide explanations, and offer guidance on various aspects of carbon credits, emissions reductions, and market mechanisms. How can I assist you today?", }

''' { "id": 1, "time": "2023-11-06T02:44:59.491757", "prompt": "What is your role?", "response": "As an AI representative of Verst Carbon, my role is to provide information and assistance regarding carbon markets and carbon credits. I can help answer any questions you may have about the role of carbon credits in achieving sustainability goals or any other related topics. How can I assist you today?", } '''

''' { "id": 1, "time": "2023-11-06T02:44:59.596009", "prompt": "What is your role?", "response": "As an AI representative of the Africa Climate Summit, my role is to provide information and support regarding the summit's objectives, deliverables, and initiatives. I can help answer any questions you may have about the summit and its focus on green growth, climate finance, and sustainable development in Africa. How can I assist you further?", } '''

[logan-markewich]

I'm not able to reproduce this. Comparing 0.8.56 to the latest, both are basically the same 🤷🏻 And I haven't seen anyone else report this in the meantime.

For reference, I ran this code 3 times with both versions, 0.8.56 had an average of 20s (20.3s, 20.6s, 19.8s), 0.8.63.post2 had an average of 20s (20.3s, 21.5s, 19.6s)

from llama_index import SimpleDirectoryReader, VectorStoreIndex, ServiceContext
from llama_index.llms import OpenAI

docs = SimpleDirectoryReader("docs/examples/data/paul_graham").load_data()

service_context = ServiceContext.from_defaults(llm=OpenAI(model="gpt-3.5-turbo", temperature=0))

index = VectorStoreIndex.from_documents(docs, service_context=service_context)

engine = index.as_chat_engine(chat_mode="context")
engine.chat("Hello!")

import time

start = time.time()

engine.chat("What did the author do growing up?")
engine.chat("What happened at interleaf?")
engine.chat("What happend at viaweb?")

end = time.time()

Going to close for now.

[gich2009]

Thanks @logan-markewich for taking a look. I will upgrade to 0.8.63 and run the tests. Incase of any trouble I'll let you know.

And also, thanks for the compartibility updates with the openai models.