search

redhat-et / foundation-models-for-documentation

Improve ROSA customer experience (and customer retention) by leveraging foundation models to do “gpt-chat” style search of Red Hat customer documentation assets.
Other
26 stars 12 forks source link

Add notebook with langchain and openai #15

Closed Shreyanand closed 1 year ago

Shreyanand commented 1 year ago

Related to #9 This PR adds a notebook that uses openai embedding and Langchain framework for the QA task.

review-notebook-app[bot] commented 1 year ago

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

Shreyanand commented 1 year ago

@codificat @suppathak Are there things that I can add to this notebook that would help with #24? If not we could start on that notebook...

codificat commented 1 year ago

I think you could probably add some more markdown cells explaining what is going on and what choices are being made among the available options. Also it would be nice to include links to relevant docs where appropriate.

Sounds good, will add that!

The evaluation approach here, using an LLM for evaluation, differs from what is presented in #16. That's fine, it is an interesting approach to evaluate, but I am wondering if it belongs there instead? Or, possibly, in the "next" notebook that you mention, #24.

Yes, the metrics notebook #16 should include a description of this as well. It's quite an interesting approach.

I mean: that part already helps with what we want for #24, but we probably don't want to elaborate much more on that here: the goal of this notebook should be to highlight what langchain offers, similar to what the hugging face and haystack notebooks did.

Agreed, so I'll split this notebook into two. The first one being a small notebook showing langchain and openai with appropriate links and markdown. The second one being a WIP notebook for #24.

Shreyanand commented 1 year ago

That one question answer looks interesting:

Query: How can I federate metrics to a centralized Prometheus Cluster?  
 Answer:  To federate metrics to a centralized Prometheus Cluster, you can deploy a Prometheus instance and configure it to send alerts to themselves, and use Prometheus' federated metrics feature and the Prometheus Operator. 
SOURCES:  
Tommer Amber's guide: https://medium.com/@tamber/2-mini-how-to-guides-for-prometheus-on-openshift-federation-custom-infrastructure-alerting-8ec70061405d

so, it is pointing to an external source, not to any of the docs provided.

I believe it would be helpful to get some more insight into what is going on here, i.e. to be able to observe which documents have been identified as relevant during the chain, their contents (after being split), what prompts (including query and context) are being sent, etc.

Yes! I skimmed over the medium article and it seemed to be relevant (tbh, I was happy the link even worked :D) I'll look at it in more detail in the demo notebook.

But this would be a separate issue.

This one LGTM