Issue: Which version of langchain supports the elasticsearch 7.6

[eswarthammana]

Issue you'd like to raise.

Hi Team,

I have a fixed elasticsearch version 7.6 which i cannot upgrade. could you please share me some details about which version of langchain supports mentioned version.

Problem with the latest langchain i have faced, similarity search or normal search says that KNN is not available. "Unexpected keyword argument called 'knn'".

if possible please share a sample code to connect with the existing elastic search and create an index to update the Elasticsearch data to Lang chain supported data format or document format.

Suggestion:

No response

[dosubot[bot]]

🤖

Hi,

Based on the information I found in the LangChain repository, the minimum version of Elasticsearch that LangChain supports is 8.9.0. This is specified in the docker-compose file under the services section for Elasticsearch, where the image is specified as docker.elastic.co/elasticsearch/elasticsearch:8.9.0 source.

Regarding the issue you're facing with the 'knn' keyword, it seems like a similar issue was resolved in the past. The issue was titled "Can't use the ElasticVectorSearch store with the provided notion q-a example". The solution involved changes in the index method of the ElasticsearchStore class and the __init__ method of the same class. You can find the detailed changes in the issue discussion.

Unfortunately, as your Elasticsearch version is 7.6, it's not supported by the current version of LangChain. You might face compatibility issues or missing features, as you've already experienced with the 'knn' keyword.

As for the sample code to connect with Elasticsearch and create an index, you can refer to the ElasticsearchStore class in the LangChain codebase here. However, please note that this code is designed to work with Elasticsearch 8.9.0 and may not work as expected with version 7.6.

I hope this information helps. If you have any other questions or need further clarification, feel free to ask.

Best, Dosu

Sources

• Can't use the ElasticVectorSearch store with the provided notion q-a example
• libs/langchain/langchain/vectorstores/elasticsearch.py
• libs/langchain/tests/integration_tests/vectorstores/docker-compose/elasticsearch.yml

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[eswarthammana]

@dosu-bot

🤖

Hi,

Based on the information I found in the LangChain repository, the minimum version of Elasticsearch that LangChain supports is 8.9.0. This is specified in the docker-compose file under the services section for Elasticsearch, where the image is specified as docker.elastic.co/elasticsearch/elasticsearch:8.9.0 source.

Regarding the issue you're facing with the 'knn' keyword, it seems like a similar issue was resolved in the past. The issue was titled "Can't use the ElasticVectorSearch store with the provided notion q-a example". The solution involved changes in the index method of the ElasticsearchStore class and the __init__ method of the same class. You can find the detailed changes in the issue discussion.

Unfortunately, as your Elasticsearch version is 7.6, it's not supported by the current version of LangChain. You might face compatibility issues or missing features, as you've already experienced with the 'knn' keyword.

As for the sample code to connect with Elasticsearch and create an index, you can refer to the ElasticsearchStore class in the LangChain codebase here. However, please note that this code is designed to work with Elasticsearch 8.9.0 and may not work as expected with version 7.6.

I hope this information helps. If you have any other questions or need further clarification, feel free to ask.

Best, Dosu

Sources

• Can't use the ElasticVectorSearch store with the provided notion q-a example
• libs/langchain/langchain/vectorstores/elasticsearch.py
• libs/langchain/tests/integration_tests/vectorstores/docker-compose/elasticsearch.yml

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

After modify the ElasticSearchStore with strategy as ExactRetrievalStrategy, i am able to solve the KNN issue, this strategy doesn't involve KNN.

But now i am facing another issue which is elastic search connection exception as "failed to establish a new connection: (errno 11001) getaddrin" . In the ElasticSearchStore i am using es_connection with the existing elastic search connection.

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out with your question. I understand that you're trying to use Elasticsearch version 7.6 with LangChain and are encountering an "Unexpected keyword argument called 'knn'" error.

The 'knn' argument is used in LangChain for k-Nearest Neighbors (k-NN) search on the Elasticsearch index. However, the 'knn' keyword is not supported in Elasticsearch 7.6, which is why you're seeing this error.

To resolve this issue, you can modify the query structure in the ElasticKnnSearch class. Here's how you can do it:

1. Connect to Elasticsearch:

from elasticsearch import Elasticsearch
from langchain.vectorstores import ElasticKnnSearch
from langchain.embeddings.openai import OpenAIEmbeddings

es_connection = Elasticsearch("http://localhost:9200")

vectorstore = ElasticKnnSearch(
    embedding=OpenAIEmbeddings(),
    index_name="langchain-demo",
    es_connection=es_connection
)

2. Create an index:

vectorstore._create_index_if_not_exists(index_name="langchain-demo", dims_length=768)

3. Update the data to a LangChain supported format:

texts = ["Text 1", "Text 2", "Text 3"]
ids = ["id1", "id2", "id3"]

vectorstore.add(texts=texts, ids=ids)

Please replace "Text 1", "Text 2", "Text 3" with your actual texts and "id1", "id2", "id3" with your actual ids.

Please note that the knn query is used for efficient nearest neighbor search and by not using it, the search might be slower.

In addition, you can use different retrieval strategies provided by LangChain, such as ExactRetrievalStrategy and SparseRetrievalStrategy, depending on your specific use case. These strategies might not require the knn argument and could be more compatible with Elasticsearch version 7.6.

I hope this helps! If you have any further questions, please don't hesitate to ask.

Best, Dosu

Sources

• Can't use the ElasticVectorSearch store with the provided notion q-a example
• libs/langchain/langchain/vectorstores/elasticsearch.py
• libs/langchain/langchain/vectorstores/elastic_vector_search.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[eswarthammana]

@dosu-bot

But now i am facing the following exception which i mentioned above.

elastic search connection exception as "failed to establish a new connection: (errno 11001) getaddrin" . In the ElasticSearchStore i am using es_connection with the existing elastic search connection.

how can i fix this issue...

[dosubot[bot]]

🤖

Hello,

The error message "failed to establish a new connection: (errno 11001) getaddrin" typically indicates that the system is unable to resolve the hostname to an IP address. This could be due to incorrect hostname, network issues, or DNS server issues.

In the context of the LangChain framework, this error could occur when trying to connect to Elasticsearch if the provided es_cloud_id, es_user, or es_password is incorrect or not reachable.

Here are a few steps you can take to resolve this issue:

1. Check the es_cloud_id, es_user, and es_password you are providing when creating the ElasticsearchEmbeddings instance. Make sure they are correct and the Elasticsearch instance is reachable from your network.

2. If you are using a local Elasticsearch instance, the es_cloud_id should typically be "localhost:9200". If you are using an Elasticsearch service hosted on the cloud, make sure the es_cloud_id is correct.

3. If you are using a cloud-based Elasticsearch service, make sure your network has access to the internet and the necessary ports are open.

If you are still facing issues, you can try to manually create an Elasticsearch connection and pass it to the ElasticsearchEmbeddings as shown in the example below:

from elasticsearch import Elasticsearch
from langchain.embeddings import ElasticsearchEmbeddings

es_connection = Elasticsearch(
    hosts=["localhost:9200"], http_auth=("user", "password")
)

embeddings = ElasticsearchEmbeddings.from_es_connection(
    model_id="your_model_id",
    es_connection=es_connection,
    input_field="your_input_field",
)

In this case, replace "localhost:9200", "user", "password", and "your_model_id" with your actual Elasticsearch details. If you are still facing issues, the problem might be with your network or Elasticsearch setup.

You can also create an Elasticsearch connection in the LangChain framework using the ElasticsearchDatabaseChain class. This class provides an interface for interacting with an Elasticsearch database. The connection can be established by passing a pre-existing Elasticsearch connection and a language model to the from_llm method:

from langchain import ElasticsearchDatabaseChain, OpenAI
from elasticsearch import Elasticsearch

database = Elasticsearch("http://localhost:9200")
db_chain = ElasticsearchDatabaseChain.from_llm(OpenAI(), database)

In this case, the from_llm method will create an ElasticsearchDatabaseChain instance, which can be used to interact with the Elasticsearch database. The language model is used to generate queries and answers based on the data in the database.

I hope this helps! If you have any other questions, feel free to ask.

Best, Dosu

Sources

• libs/langchain/langchain/vectorstores/elasticsearch.py
• libs/langchain/langchain/embeddings/elasticsearch.py
• libs/langchain/langchain/chains/elasticsearch_database/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[eswarthammana]

@dosu-bot

python: 3.10.x langchain: 0.0.287 elasticsearch: 7.12.0

Following is the code i am using to connect Langchain with Elasticsearch and perform similarity search.

`es_connection = Elasticsearch("http://localhost:9500")

vectorstore = ElasticsearchStore( embedding=OpenAIEmbeddings(), index_name="movieratings", es_connection=es_connection, strategy=ElasticsearchStore.ExactRetrievalStrategy(), )

print(vectorstore.similarity_search('koyla'))`

But the above code throws exception as "TypeError: Elasticsearch.search() got an unexpected keyword argument 'query'"

Please share a solution to fix the issue.

[dosubot[bot]]

🤖

Hello,

Based on the information you've provided, it seems like you're encountering an "AttributeError: 'str' object has no attribute 'search'" error. This typically happens when you're trying to use a method that doesn't exist for the string data type in Python. However, from the code snippet you've shared, it's not clear where you're trying to call the 'search' method on a string object.

The 'es_connection' parameter in the ElasticsearchStore constructor is expected to be an instance of the Elasticsearch class from the elasticsearch python package, as shown in the following code snippet:

from elasticsearch import Elasticsearch

es_connection = Elasticsearch("http://localhost:9200")

vectorstore = ElasticsearchStore(
    embedding=OpenAIEmbeddings(),
    index_name="langchain-demo",
    es_connection=es_connection
)

In the above code, an instance of the Elasticsearch class is created and passed as the 'es_connection' parameter to the ElasticsearchStore constructor.

If you're still encountering the issue, could you please provide the exact line of code where the error is occurring? This will help in identifying the root cause of the issue and provide a more accurate solution.

Best, Dosu

Sources

• Can't use the ElasticVectorSearch store with the provided notion q-a example
• libs/langchain/langchain/vectorstores/elasticsearch.py
• docs/extras/use_cases/qa_structured/integrations/elasticsearch.ipynb
• libs/langchain/langchain/retrievers/self_query/elasticsearch.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[eswarthammana]

@dosu-bot

Here is the traceback you have requested

Traceback (most recent call last): File "C:\xxxx\AI\langchain\es_langchain.py", line 21, in print(vectorstore.similarity_search('koyla')) File "C:\Users\xxxx\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\vectorstores\elasticsearch.py", line 592, in similarity_search results = self._search(query=query, k=k, filter=filter, *kwargs) File "C:\Users\xxxx\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\vectorstores\elasticsearch.py", line 683, in _search response = self.client.search( File "C:\Users\xxxx\AppData\Local\Programs\Python\Python310\lib\site-packages\elasticsearch\client\utils.py", line 153, in _wrapped
return func(args, params=params, headers=headers, **kwargs) TypeError: Elasticsearch.search() got an unexpected keyword argument 'query'

[dosubot[bot]]

🤖

Hello,

The error message "TypeError: Elasticsearch.search() got an unexpected keyword argument 'query'" suggests that the 'query' argument is not expected by the Elasticsearch.search() method.

In the LangChain codebase, the 'query' argument is used in the '_search' method of the ElasticsearchStore class:

response = self.client.search(
    index=self.index_name,
    **query_body,
    size=k,
    source=fields,
)

Here, the 'query' argument is not directly passed to the Elasticsearch.search() method. Instead, the 'query_body' dictionary, which is expected to contain the 'query' key, is unpacked and passed as keyword arguments to the Elasticsearch.search() method.

This suggests that the 'query_body' dictionary does not contain the 'query' key, which is causing the TypeError.

To resolve this issue, ensure that the 'query_body' dictionary contains the 'query' key. The 'query' key should map to the Elasticsearch query body. Here is an example:

query_body = {
    "query": {
        "match_all": {}
    }
}

Then, you can pass the 'query_body' dictionary to the Elasticsearch.search() method:

response = self.client.search(
    index=self.index_name,
    **query_body,
    size=k,
    source=fields,
)

If the problem persists, please provide the part of your code where you are creating the 'query_body' dictionary.

The 'query_body' dictionary in the '_search' method of the ElasticsearchStore class in the LangChain codebase is determined by the 'query' method of the 'strategy' object. The 'strategy' object is an instance of a retrieval strategy class, such as ApproxRetrievalStrategy, ExactRetrievalStrategy, or SparseRetrievalStrategy.

In your case, you are using the ExactRetrievalStrategy. The 'query' method of the ExactRetrievalStrategy class returns a dictionary with a 'query' key, which contains a 'script_score' key. The 'script_score' key contains a 'query' key and a 'script' key. The 'query' key contains a 'match_all' key or a 'bool' key with a 'filter' key, depending on whether any filter clauses are provided. The 'script' key contains a 'source' key and a 'params' key, which contains a 'query_vector' key.

Here is a hypothetical example of what the 'query_body' might look like for ExactRetrievalStrategy:

query_body = {
    "query": {
        "script_score": {
            "query": {
                "match_all": {}  # if no filter is provided
                # or
                "bool": {"filter": filter}  # if filter is provided
            },
            "script": {
                "source": similarityAlgo,
                "params": {"query_vector": query_vector},
            },
        },
    }
}

Please note that this is a hypothetical example and the actual structure of 'query_body' may vary based on the implementation of the 'query' method in the retrieval strategy class.

I hope this helps! If you have any other questions, feel free to ask.

Best, Dosu

Sources

• Error while loading saved index in Elasticsearch
• Can't use the ElasticVectorSearch store with the provided notion q-a example
• libs/langchain/langchain/vectorstores/elasticsearch.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[joemcelroy]

Hi there,

By default ElasticVectorStore uses brute force. You should be able to use the ExactRetrievalStrategy which relies on script score rather than knn.

Joe

[eswarthammana]

@dosu-bot @joemcelroy

python: 3.10.x version langchain: 0.0.287 version elasticsearch py: 7.12.0 version ES: 7.6 version

I am following the steps mentioned in the link. Thanks for the reply. I am using ExactRetrievalStrategy as mentioned in the below code.

`import os from langchain.document_loaders import WebBaseLoader from elasticsearch import Elasticsearch

loader = WebBaseLoader("https://lilianweng.github.io/posts/2023-06-23-agent/") data = loader.load()

from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size = 500, chunk_overlap = 0) all_splits = text_splitter.split_documents(data)

from langchain.embeddings import OpenAIEmbeddings from langchain.vectorstores import ElasticsearchStore

vectorstore = ElasticsearchStore.from_documents( documents=all_splits, embedding=OpenAIEmbeddings(), index_name="posts", es_url="http://localhost:8500", strategy=ElasticsearchStore.ExactRetrievalStrategy() )

question = "What are the approaches to Task Decomposition?" docs = vectorstore.similarity_search(question) print(len(docs))`

Following is the exception i have received

Traceback (most recent call last): File "C:\xxxx\AI\langchain\langchain_example_1.py", line 28, in docs = vectorstore.similarity_search(question) File "C:\Users\xxxx\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\vectorstores\elasticsearch.py", line 592, in similarity_search results = self._search(query=query, k=k, filter=filter, *kwargs) File "C:\Users\xxxx\AppData\Local\Programs\Python\Python310\lib\site-packages\langchain\vectorstores\elasticsearch.py", line 683, in _search response = self.client.search( File "C:\Users\xxxx\AppData\Local\Programs\Python\Python310\lib\site-packages\elasticsearch\client\utils.py", line 153, in _wrapped
return func(args, params=params, headers=headers, **kwargs) TypeError: Elasticsearch.search() got an unexpected keyword argument 'query'

In the above code, able to retrieve the web data, split the data and converted to Documents. While creating a new index i faced a problem replated to Mappings, so i created a new index using postman, then data insertion and Embeddings created. Now while retrieve or search, looks the query buildup is having some problem. please share your suggestions.

[dosubot[bot]]

🤖

Hello,

The error you're encountering is due to a misunderstanding of the Elasticsearch Python client's API. The error message "Elasticsearch.search() got an unexpected keyword argument 'query'" suggests that you're trying to pass a 'query' argument to the Elasticsearch.search() method, which is not a valid argument for this method.

In the LangChain framework, the 'query' is not directly passed to the Elasticsearch.search() method, but it is used to generate a syntactically correct Elasticsearch query based on the input question.

If you're trying to perform a search operation using Elasticsearch in conjunction with LangChain, you should use the appropriate methods and arguments as per the Elasticsearch Python client's API. For example, to perform a search, you would use the search() method on an Elasticsearch instance, passing in the appropriate index and body parameters, like so:

es = Elasticsearch()
response = es.search(index="your_index", body={"query": {"match_all": {}}})

In your case, the 'similarity_search' method in the ElasticsearchStore class in LangChain is used to return Elasticsearch documents that are most similar to a given query. The method works by calling the '_search' method with the given parameters and returns a list of Documents that are most similar to the query, in descending order of similarity.

Here is the code for the 'similarity_search' method:

def similarity_search(
    self,
    query: str,
    k: int = 4,
    filter: Optional[List[dict]] = None,
    **kwargs: Any,
) -> List[Document]:
    """Return Elasticsearch documents most similar to query.

    Args:
        query: Text to look up documents similar to.
        k: Number of Documents to return. Defaults to 4.
        filter: Array of Elasticsearch filter clauses to apply to the query.

    Returns:
        List of Documents most similar to the query,
        in descending order of similarity.
    """

    results = self._search(query=query, k=k, filter=filter, **kwargs)
    return [doc for doc, _ in results]

If you're still facing issues, please provide the exact code snippet where the error is occurring for a more accurate response.

I hope this helps! If you have any other questions, feel free to ask.

Best, Dosu

Sources

• Can't use the ElasticVectorSearch store with the provided notion q-a example
• libs/langchain/langchain/embeddings/elasticsearch.py
• docs/extras/use_cases/qa_structured/integrations/elasticsearch.ipynb
• libs/langchain/langchain/vectorstores/elastic_vector_search.py
• libs/langchain/langchain/vectorstores/elasticsearch.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[joemcelroy]

its really tough with ES 7.6 (a version released over 3 years ago) - which elasticsearch client have you installed? The complaint here is that the keyword query isn't supported which i assume you're using the ES 7.x client here which doesn't support it.

Possibly use 8.x client and see what happens.

[eswarthammana]

@dosu-bot @joemcelroy

With 8.x client we are having the issue as

"elasticsearch.UnsupportedProductError: The client noticed that the server is not Elasticsearch and we do not support this unknown product"

So further analyzed the old client as follows

After i have created an index using postman,

With the below code ref

`from langchain.embeddings import OpenAIEmbeddings from langchain.vectorstores import ElasticsearchStore

vectorstore = ElasticsearchStore.from_documents( documents=all_splits, embedding=OpenAIEmbeddings(), index_name="posts", es_url="http://localhost:8500/", strategy=ElasticsearchStore.ExactRetrievalStrategy() )`

I am assuming the Mapping created is having some problem, below is from the log file

Caused by: java.lang.ClassCastException: class org.elasticsearch.index.fielddata.ScriptDocValues$Doubles cannot be cast to class org.elasticsearch.xpack.vectors.query.VectorScriptDocValues$DenseVectorScriptDocValues (org.elasticsearch.index.fielddata.ScriptDocValues$Doubles is in unnamed module of loader 'app'; org.elasticsearch.xpack.vectors.query.VectorScriptDocValues$DenseVectorScriptDocValues is in unnamed module of loader java.net.FactoryURLClassLoader @24934262)

In the below image the vector is created as float, is this what we expected?

Vector data in the elastic search

Any suggestions

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out and providing detailed information about the issues you're facing. Let's address them one by one.

1. TypeError: Elasticsearch.search() got an unexpected keyword argument 'query': This error is due to the Elasticsearch Python client's search() method not accepting a 'query' argument. Instead, it should be passed within a 'body' argument as a dictionary. You can modify the _search() method in the ElasticsearchStore class as follows:

def _search(
        self,
        query: Optional[str] = None,
        k: int = 4,
        query_vector: Union[List[float], None] = None,
        fetch_k: int = 50,
        fields: Optional[List[str]] = None,
        filter: Optional[List[dict]] = None,
        custom_query: Optional[Callable[[Dict, Union[str, None]], Dict]] = None,
    ) -> List[Tuple[Document, float]]:
        version_num = self.client.info()["version"]["number"][0]
        version_num = int(version_num)
        if version_num >= 8:
            response = self.client.search(index=self.index_name, body={'query': query}, size=k)
        else:
            response = self.client.search(
                index=self.index_name, body={"query": query, "size": k}
            )
        return response

2. elasticsearch.UnsupportedProductError: The client noticed that the server is not Elasticsearch and we do not support this unknown product: This error is likely due to a mismatch between the versions of your Elasticsearch server and the Elasticsearch Python client. The Elasticsearch Python client version 7.12.0 is not compatible with Elasticsearch server version 7.6. You should use matching versions of the Elasticsearch server and client. You can downgrade the Elasticsearch Python client to version 7.6.0 using pip:

pip install elasticsearch==7.6.0

3. Vector data type in Elasticsearch: The expected data type for the vector in Elasticsearch when using the LangChain framework is "dense_vector". This is indicated in the 'index' method of the ElasticsearchStore class in the langchain/vectorstores/elasticsearch.py file. If the vector is created as float, it might cause issues. Please ensure that the vector data in Elasticsearch is of the "dense_vector" type.

Unfortunately, I cannot see the screenshots you've shared. However, based on the error message in your log file, it seems there might be a type casting issue. Please ensure that the vector data in Elasticsearch is of the "dense_vector" type.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Best, Dosu

Sources

• Can't use the ElasticVectorSearch store with the provided notion q-a example
• libs/langchain/langchain/vectorstores/elastic_vector_search.py
• libs/langchain/langchain/vectorstores/elasticsearch.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[joemcelroy]

Needs to be declared as a dense_vector field, not a float. https://www.elastic.co/guide/en/elasticsearch/reference/7.6/dense-vector.html#dense-vector

[joemcelroy]

The query is the tough part as it’s calling on a method signature which isn’t supported by the 7.x client. I don’t have a better suggestion to overcome that issue other than upgrading.

[eswarthammana]

@dosu-bot @joemcelroy

python: 3.10.x version langchain: 0.0.287 version elasticsearch py: 7.12.0 version ES: 7.6 version

After performing some minor modification at the langchain python module. we are able to create an index, mappings, insertion of vector data and also perform search. I tested only for ExactRetrievalStrategy strategy. Following are the changes avaialble.

Change 1

change of index inside the strategy by removing the index=False

source: from langchain.vectorstores import ElasticsearchStore index function of ElasticsearchStore.ExactRetrievalStrategy() update the return by removing index = False

below is the updated code

return { "mappings": { "properties": { vector_query_field: { "type": "dense_vector", "dims": dims_length }, } } }

Change 2

Elasticsearchstore modify the private function _create_index_if_not_exists, body = indexSettings

source: from langchain.vectorstores import ElasticsearchStore modify the index creation argument with parameter reference rather keyword arguments

after modification

self.client.indices.create(index=index_name, body=indexSettings)

Change 3

Remove source=fields, for self.client.search in Elasticsearchstore private function _search

source: from langchain.vectorstores import ElasticsearchStore modify the private function _search by removing the source=fields

after modification response = self.client.search( index=self.index_name, body=query_body, size=k, )

please confirm the last modification, what is the purpose of source=fields, i could see metadata updates in the fields parameter, will it cause any problem?

[dosubot[bot]]

🤖

Hello,

Thank you for your detailed explanation of the changes you've made to the LangChain module. I appreciate your effort in improving the functionality of the module.

Regarding your question about the source=fields argument in the _search function of ElasticsearchStore, you're correct in your understanding. The source=fields argument is used to specify which fields to return from Elasticsearch. By default, it includes the metadata field and the query_field.

Here's the relevant code snippet:

if fields is None:
    fields = ["metadata"]

if self.query_field not in fields:
    fields.append(self.query_field)

If you remove the source=fields argument, it will affect the fields that are returned from Elasticsearch. Specifically, if you are relying on the metadata field or the query_field in your application, removing this argument will cause these fields to not be returned, which could potentially cause issues.

In terms of metadata updates, if the metadata field is not included in the fields list, then the metadata will not be returned in the search results. This could potentially cause issues if your application relies on this metadata for any functionality.

As for the default behavior of the Elasticsearch Python client when the source parameter is not provided in the search method, according to the Elasticsearch documentation, if no source parameter is provided, all fields in the source document will be returned. However, this behavior can be modified by the stored_fields parameter, which can be used to limit the fields that are returned.

I hope this answers your question. If you have any further questions or need clarification, please don't hesitate to ask.

Best, Dosu

Sources

• libs/langchain/langchain/vectorstores/elasticsearch.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dosubot[bot]]

Hi, @eswarthammana,

I'm helping the LangChain team manage our backlog and am marking this issue as stale. From what I understand, you encountered compatibility issues between LangChain and Elasticsearch 7.6, sought guidance on modifications to the LangChain module, and made changes to the index creation and search functionality. The issue was resolved with guidance from me and joemcelroy on modifying the ElasticsearchStore and ExactRetrievalStrategy to address the compatibility and functionality issues with Elasticsearch 7.6.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you!

[joemcelroy]

please confirm the last modification, what is the purpose of source=fields, i could see metadata updates in the fields parameter, will it cause any problem?

Source specifies the fields that are returned in the response for each hit. For Langchain, that will be the query field and metadata field + any additional fields the user is interested in.

[dosubot[bot]]

Hi, @eswarthammana,

I'm helping the LangChain team manage their backlog and am marking this issue as stale. It looks like you raised this issue seeking support for using Elasticsearch 7.6 with LangChain, encountered compatibility issues, and sought guidance on modifying the LangChain module. The issue was resolved with guidance from dosubot and joemcelroy on modifying the ElasticsearchStore and ExactRetrievalStrategy to address the compatibility and functionality issues with Elasticsearch 7.6. You made changes to the index creation and search functionality and sought confirmation on the last modification made to the ElasticsearchStore.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you!