posts/semantic-search-elasticsearch-openai-langchain/

[utterances-bot]

Semantic Search with Elasticsearch, OpenAI, and LangChain

In this tutorial, I'll walk you through building a semantic search service using Elasticsearch, OpenAI, LangChain, and FastAPI. You'll create an application that lets users ask questions about Marcus Aurelius' Meditations and provides them with concise answers by extracting the most relevant content from the book.

https://dylancastillo.co/semantic-search-elasticsearch-openai-langchain/

[arieefrachman]

Hi, could you please give an example to provide a document metadata while query the content?

[dylanjcastillo]

@arieefrachman You can use return_source_documents=True when you initialize the chain to get the source document when using the chain.

[shrimantasatpati]

elasticsearch_url="http://localhost:9200" How can I make use of it as I am getting the error connection request to elastic search is failed when I run the indexer.py. Can you kindly tell me a detailed way to set the elastic search locally or using the elastic search cloud?

[dylanjcastillo]

@shrimantasatpati are you sure you're running elasticsearch locally? I'd need more info about the error code to help you

[shrimantasatpati]

elastic_transport.ConnectionError: Connection error caused by: ConnectionError(Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x0000023D90B43D10>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it))

[dylanjcastillo]

Are you sure you're running Elasticsearch locally?

[udnscott]

I have different issue: https://stackoverflow.com/questions/76904020/is-it-possible-to-use-langchain-to-chat-with-in-house-elasticsearch-engine

Can I do it and how?

[dylanjcastillo]

@udnscott yes, you can. You don't need to create a new index, you can simply use the results from the previous engine and add those to the context.

Look at this section from a previous tutorial. You just need to replace similar_docs with the search results from your Elasticsearch instance.

What you cannot do is semantic search without re-indexing the data.

[jlia0]

Thanks for the great article!! I have a few questions...

For the suggestion - "Build appropriate data ingestion and processing pipelines. When adding new data, you generally do not want to rebuild the index, as you did in this tutorial."

What is the best practice for this? If I need to keep adding new documents, don't I need to rebuild the indexes every time? And how do I access control/filtering indexes?