feature request - add chat history

[Hisma]

the current version, every time a query is ran, the previous query is overrwriten. It is not too difficult to add chat history. via the streamlit api. You can simply use the session_state.messages.append method to have previous messages appended to the chat window, rather than erased on every new query.

Here's what I hacked together -

  ` 
    # Add user message to chat history
    st.session_state.messages.append({"role": "user", "content": text})

    # Add assistant response to chat history
    st.session_state.messages.append({"role": "assistant", "content": output.response})
    st.session_state.messages.append({"role": "assistant", "content": f"***sources :*** {source_links}"})
    st.session_state.messages.append({"role": "assistant", "content": f"**Search results quality score: {output.average_score:.2f}**\n"})`

However, I am not a skilled programmer, and while it does work, I am sure there's a more elegant way to do this.

Can this please be added?

[snexus]

Thanks for the suggestion - good idea. Will add it to the next release. Guess historical source chunks (as implied by your code snippet) should be excluded from the history, as output would be overwhelming after a few runs.

[Hisma]

Thank you! I created a list that only stores the links, like this. Here's the full snippet of code -

`

Display user message in chat container

    with chat_message("user"):
        st.markdown(text)
    with chat_message("assistant"):
         source_links = []
         st.write(output.response)
         for source in output.semantic_search[::-1]:
             source_links.append(source.chunk_link)
             source_path = source.metadata.pop("source")
             score = source.metadata.get("score", None)
             with st.expander(label=f":file_folder: {source_path}", expanded=False):
                 st.write(
                     f'<a href="{source.chunk_link}">{source.chunk_link}</a>',
                     unsafe_allow_html=True,
                 )
                 if score is not None:
                     st.write(f"\nScore: {score}")
                 st.text(f"\n\n{source.chunk_text}")
         st.write(f"**Search results quality score: {output.average_score:.2f}**\n")`

So as you can see, I left most of your original code intact, but I added a list to store the deep links of the sources, and then append the links to the list.
Then the history only keeps tracks of the source deep links, rather than the entire source chunk.
It's just a little ugly because it prints a list without any formatting. Would be nice if it could print something like source 1 - deep link, source 2 - deep link, source n - deep link, with the links being clickable hyperlinks.
I think that would really improve the UI/UX!

[snexus]

I've pushed an initial version to the feature branch - https://github.com/snexus/llm-search/tree/feature-web-chat-history

Not sure if it is the best way from the UI perspective, but it works - the history is added at the top as expandable cards, which contain information from the previous queries (with clickable links). Please let me know what you think.

[Hisma]

hi, so I tested this version with the chat history. The implementation is excellent! I like that it's collapsed by default and I can expand it as I need to. I did encounter a couple issues

• the quality of my search results has somehow gone down. I tested by asking the llm the same exact questions I asked it in the previous version, and this time I am being told that "there is not a direct answer to your question in the context provided". Strange. Note that I also have the hyde updated code.
• Speaking of hyde, when I toggle hyde on or off, it clears the chat history completely. However, when I ask another question, the chat history re-appears.

I'm concerned about why I'm getting different responses to the same questions using the same exact models. I wonder if the changes you made with the hyde update perhaps made the responses try to be more precise, and because it thinks it doesn't have an EXACT answer it refuses to give a response?

This is separate from the chat history issue. However, I am keen to perhaps roll back my chat app to the previous version until we can explain what caused the app to stop finding answers in the context given... note this is with hyde option on or off.

Anyway, to close this particular topic, you'll need to fix that bug with the chat history disappearing when the hyde option is toggled. Other than that, great job!

[Hisma]

also, it appears that while it does keep history of previous questions & responses, when you ask a new question, it doesn't show the question, only the response.

[Hisma]

sorry for the spam, but I went through and looked at the code changes between the different commits. Looks like the changes you made only really added the HyDE capability and not much else was touched, aside from adding the history feature.

So I can't really make sense as to why my model got "dumber" after doing this update. By the way, I'm using wizard 13B via llama.cpp. I'm going to try some other models and see if I can get some better results.

OpenAI did seem to be better for what it's worth. I'll need to test further. But I can say 100% that the wizard-13B model reacted differently (in a negative way) when the code for hyde was added.

[snexus]

Thank you for the feedback. Deterioration of results is indeed concerning - I am not quite sure why it happens: if HyDE is turned off, there should be no change to the user's query and the result should remain the same. Turning HyDE on can definitely change the results and should be used with caution.

If you don't mind, would you be able to test your queries for the previous version vs the new version (with HyDE turned off)? Please make sure the query is identical between the versions - smaller models are quite sensitive even to slight changes, such as capitalization.

I wonder if the changes you made with the hyde update perhaps made the responses try to be more precise

This is actually controlled by the user in the config file via prompt engineering, there is no additional postprocessing happening to ensure the preciseness of the results.

Another question - did you change your environment in any way? E.g. upgraded llama cpp version or reinstalled the package completely within a new environment? (which would pull the new llama cpp, it is not pinned at the moment)

[Hisma]

I have isolated the problem down to turning the HyDE feature on in the config.yaml file. So it's definitely using the HyDE feature that's making the model dumber. Here is the results.

HyDE feature commented out in config.yaml - This is a proper response.

console output - 2023-10-19 22:01:15.991 | INFO | llmsearch.embeddings:get_embedding_model:30 - Embedding model config: type=<EmbeddingModelType.sentence_transformer: 'sentence_transformer'> model_name='BAAI/bge-large-en-v1.5' additional_kwargs={} 2023-10-19 22:01:17.460 | INFO | llmsearch.ranking:init:18 - Initializing Reranker... 2023-10-19 22:01:17.764 | INFO | llmsearch.splade:init:37 - Setting device to cuda:0 2023-10-19 22:01:18.477 | INFO | llmsearch.splade:load:111 - SPLADE: Got 1 labels. 2023-10-19 22:01:18.477 | INFO | llmsearch.splade:load:118 - Loaded sparse (SPLADE) embeddings from /home/hisma/rag/embeddings/splade/splade_embeddings.npz 2023-10-19 22:01:18.480 | INFO | llmsearch.utils:get_llm_bundle:78 - Initialized persistence db. 2023-10-19 22:01:18.480 | INFO | llmsearch.utils:get_hyde_chain:97 - Creating HyDE chain... 2023-10-19 22:01:31.178 | INFO | llmsearch.ranking:get_relevant_documents:46 - Adding query prefix for retrieval: query: 2023-10-19 22:01:31.178 | INFO | llmsearch.splade:query:186 - SPLADE search will search over all documents of chunk size: 1024. Number of docs: 3045 [0.13744058 0.17504036 0.21993668 ... 0.12635042 0.0828779 0.0014901 ] 2023-10-19 22:01:31.219 | INFO | llmsearch.ranking:get_relevant_documents:61 - Stage 1: Got 15 documents. 2023-10-19 22:01:31.219 | INFO | llmsearch.ranking:get_relevant_documents:73 - Dense embeddings filter: None 2023-10-19 22:01:31.610 | INFO | llmsearch.ranking:rerank:23 - Reranking documents ... 2023-10-19 22:01:31.660 | INFO | llmsearch.ranking:rerank:32 - [2.3267693519592285, 1.583428978919983, 1.4721457958221436, 0.9861947298049927, 0.8727389574050903, 0.7136256694793701, 0.5252503156661987, 0.45489561557769775, 0.39229536056518555, 0.3083770275115967, 0.18841665983200073, -0.08134979009628296, -0.19731353223323822, -0.3803035318851471, -0.3900640606880188, -0.7138215899467468, -0.8032354712486267, -1.1959598064422607, -1.4357192516326904, -2.277240753173828, -4.254514694213867, -4.635542392730713, -4.82810115814209, -4.947267532348633, -4.962316513061523, -5.178018569946289, -6.359096527099609, -8.98660659790039] 2023-10-19 22:01:31.660 | INFO | llmsearch.ranking:get_relevant_documents:96 - Number of documents after stage 1 (sparse): 15 2023-10-19 22:01:31.660 | INFO | llmsearch.ranking:get_relevant_documents:99 - Number of documents after stage 2 (dense + sparse): 28 2023-10-19 22:01:31.660 | INFO | llmsearch.ranking:get_relevant_documents:102 - Re-ranker avg. scores for top 5 resuls, chunk size 1024: 1.45 [chain/start] [1:chain:StuffDocumentsChain] Entering Chain run with input: [inputs] [chain/start] [1:chain:StuffDocumentsChain > 2:chain:LLMChain] Entering Chain run with input: { "question": "GHG footprint of pea protein", "context": "passage: not affected.\n5. Conclusions\nOur approach shows the potential of combining environmental\nand nutritional attributes into quantitative footprints that can be\nused to assess sustainable food options in the food sector. Our\nattributional life cycle assessment of 100 g of cooked pea protein\nballs versus beef meatballs made from Irish or Brazilian beef from\ncradle to fork showed that pea protein balls have a smaller envi-\nronmental\nimpact\nacross\nmost\nimpact\ncategories\nassessed,\nregardless of the origin of the beef or the allocation method\nemployed. Using a biophysical instead of an economic allocation\napproach across cattle co-products lowered significantly the envi-\nronmental footprint of the meatballs. Due to their higher nutrient\ndensity, the comparative environmental advantage of pea protein\nballs over beef meat balls was extended when footprints were\nexpressed per unit of nutrition.\nThe carbon opportunity cost associated with land occupation is\nusually not represented within food footprints, yet has an enor-\n\npassage: impact indicators expressed per kg of protein are presented in the\nsupplementary information and follow the same reasoning as the\nglobal warming potential.\nThe conversion from total mass to protein as a functional unit is\ninversely related to the environmental indicators due to the low\npurity of the milder fractionated ingredients. Nevertheless, a lower\ndegree of refining can still deliver the same amount of protein with\na lower footprint (Fig. 6). The carbon footprint to produce con-\nventional yellow pea protein isolate is 5.3 kg CO2-equivalents/kg\nprotein and 4.9 kg CO2-equivalents/kg protein for mild aqueous\nfractions. The fine fraction has the lowest climate change potential\nwith 1.6 kg CO2-equivalents/kg protein. The global warming po-\ntential of hybrid protein-rich fractions is 5.1 kg CO2-equivalents/kg\nprotein. The impact of the hybrid protein fractions surpasses\nthe impact of the mild aqueous protein fractions. This is due to a\nmatter of impact allocation to the protein/fibre-rich stream and the\n\npassage: rice), pseudocereal based (e.g. quinoa, amaranth, buckwheat), nut\nbased (e.g. almonds) and seed based (e.g. hemp, linseed). Of the\nexamples mentioned, only the milk alternatives made from soy,\npea and lentil proteins have protein content comparable to that\nof cowʼs milk.61\nCarbon footprints per 100 g milk alternative are reported to\nrange between 22 and 51 g CO2 equivalents for soy-based\ndrinks58,62-64 and 61 g CO2 equivalents for a pea-based drink.65\nThe carbon footprint calculated for the lentil-based prototype\ndeveloped within Protein2Food is at the lower end of the num-\nbers reported for protein-rich milk alternatives.\nThe LCA results of Protein2Food prototypes rely on assumptions\nregarding process efficiencies along the value chain and the\nachievable economic value of by-products from crop processing.\nYet the assumptions were backed by expert judgments from par-\nticipating small and medium-sized enterprises, among others,\nthereby limiting uncertainty substantially. In fact, the results pre-\n\npassage: carbon footprint of the human diet. However, the ability to convert plant proteins into functional ingredients can \nhamper their use by the food industry. Our objective was to assemble pea protein isolate (PPI) aggregates into \nedible fibers. They were obtained by co-injection of a suspension of PPI aggregates with a solution of calcium \nchloride by means of physico-chemical bonds, i.e. in the absence of thermal treatment or chemical reaction. As \nsoon as specific hydrodynamic conditions were met, homogeneous fibers of few hundreds μm diameter and few \ncm length were obtained. Small angle X-ray scattering showed that the building blocks of these fibers were dense \naggregates of 400 nm radius. Calcium and PPI concentrations required for the processing of fibers were roughly \ngiven by the sol-gel state diagram of PPI aggregates. Compared to their dairy protein based equivalent, PPI fibers \nwere obtained with a low mass fraction of protein (3% w/w) and were stable regardless of the concentration of" } [llm/start] [1:chain:StuffDocumentsChain > 2:chain:LLMChain > 3:llm:CustomLlamaLangChainModel] Entering LLM run with input: { "prompts": [ "### Instruction:\nUse the following pieces of context to answer the question at the end. If answer isn't in the context, say that you don't know, don't try to make up an answer.\n\n### Context:\n---------------\npassage: not affected.\n5. Conclusions\nOur approach shows the potential of combining environmental\nand nutritional attributes into quantitative footprints that can be\nused to assess sustainable food options in the food sector. Our\nattributional life cycle assessment of 100 g of cooked pea protein\nballs versus beef meatballs made from Irish or Brazilian beef from\ncradle to fork showed that pea protein balls have a smaller envi-\nronmental\nimpact\nacross\nmost\nimpact\ncategories\nassessed,\nregardless of the origin of the beef or the allocation method\nemployed. Using a biophysical instead of an economic allocation\napproach across cattle co-products lowered significantly the envi-\nronmental footprint of the meatballs. Due to their higher nutrient\ndensity, the comparative environmental advantage of pea protein\nballs over beef meat balls was extended when footprints were\nexpressed per unit of nutrition.\nThe carbon opportunity cost associated with land occupation is\nusually not represented within food footprints, yet has an enor-\n\npassage: impact indicators expressed per kg of protein are presented in the\nsupplementary information and follow the same reasoning as the\nglobal warming potential.\nThe conversion from total mass to protein as a functional unit is\ninversely related to the environmental indicators due to the low\npurity of the milder fractionated ingredients. Nevertheless, a lower\ndegree of refining can still deliver the same amount of protein with\na lower footprint (Fig. 6). The carbon footprint to produce con-\nventional yellow pea protein isolate is 5.3 kg CO2-equivalents/kg\nprotein and 4.9 kg CO2-equivalents/kg protein for mild aqueous\nfractions. The fine fraction has the lowest climate change potential\nwith 1.6 kg CO2-equivalents/kg protein. The global warming po-\ntential of hybrid protein-rich fractions is 5.1 kg CO2-equivalents/kg\nprotein. The impact of the hybrid protein fractions surpasses\nthe impact of the mild aqueous protein fractions. This is due to a\nmatter of impact allocation to the protein/fibre-rich stream and the\n\npassage: rice), pseudocereal based (e.g. quinoa, amaranth, buckwheat), nut\nbased (e.g. almonds) and seed based (e.g. hemp, linseed). Of the\nexamples mentioned, only the milk alternatives made from soy,\npea and lentil proteins have protein content comparable to that\nof cowʼs milk.61\nCarbon footprints per 100 g milk alternative are reported to\nrange between 22 and 51 g CO2 equivalents for soy-based\ndrinks58,62-64 and 61 g CO2 equivalents for a pea-based drink.65\nThe carbon footprint calculated for the lentil-based prototype\ndeveloped within Protein2Food is at the lower end of the num-\nbers reported for protein-rich milk alternatives.\nThe LCA results of Protein2Food prototypes rely on assumptions\nregarding process efficiencies along the value chain and the\nachievable economic value of by-products from crop processing.\nYet the assumptions were backed by expert judgments from par-\nticipating small and medium-sized enterprises, among others,\nthereby limiting uncertainty substantially. In fact, the results pre-\n\npassage: carbon footprint of the human diet. However, the ability to convert plant proteins into functional ingredients can \nhamper their use by the food industry. Our objective was to assemble pea protein isolate (PPI) aggregates into \nedible fibers. They were obtained by co-injection of a suspension of PPI aggregates with a solution of calcium \nchloride by means of physico-chemical bonds, i.e. in the absence of thermal treatment or chemical reaction. As \nsoon as specific hydrodynamic conditions were met, homogeneous fibers of few hundreds μm diameter and few \ncm length were obtained. Small angle X-ray scattering showed that the building blocks of these fibers were dense \naggregates of 400 nm radius. Calcium and PPI concentrations required for the processing of fibers were roughly \ngiven by the sol-gel state diagram of PPI aggregates. Compared to their dairy protein based equivalent, PPI fibers \nwere obtained with a low mass fraction of protein (3% w/w) and were stable regardless of the concentration of\n---------------\n\n### Question: GHG footprint of pea protein\n### Response:" ] } The GHG footprint of pea protein is approximately 1.6 kg CO2-equivalents per kilogram of protein, which is lower than that of conventional yellow pea protein isolate (5.3 kg CO2-equivalents/kg protein) and mild aqueous fractions (4.9 kg CO2-equivalents/kg protein). llama_print_timings: load time = 1270.60 ms llama_print_timings: sample time = 23.01 ms / 81 runs ( 0.28 ms per token, 3520.06 tokens per second) llama_print_timings: prompt eval time = 2864.23 ms / 1177 tokens ( 2.43 ms per token, 410.93 tokens per second) llama_print_timings: eval time = 5449.09 ms / 80 runs ( 68.11 ms per token, 14.68 tokens per second) llama_print_timings: total time = 8415.63 ms [llm/end] [1:chain:StuffDocumentsChain > 2:chain:LLMChain > 3:llm:CustomLlamaLangChainModel] [8.42s] Exiting LLM run with output: { "generations": [ [ { "text": "The GHG footprint of pea protein is approximately 1.6 kg CO2-equivalents per kilogram of protein, which is lower than that of conventional yellow pea protein isolate (5.3 kg CO2-equivalents/kg protein) and mild aqueous fractions (4.9 kg CO2-equivalents/kg protein).", "generation_info": null } ] ], "llm_output": null, "run": null } [chain/end] [1:chain:StuffDocumentsChain > 2:chain:LLMChain] [8.42s] Exiting Chain run with output: { "text": "The GHG footprint of pea protein is approximately 1.6 kg CO2-equivalents per kilogram of protein, which is lower than that of conventional yellow pea protein isolate (5.3 kg CO2-equivalents/kg protein) and mild aqueous fractions (4.9 kg CO2-equivalents/kg protein)." } [chain/end] [1:chain:StuffDocumentsChain] [8.42s] Exiting Chain run with output: { "output_text": "The GHG footprint of pea protein is approximately 1.6 kg CO2-equivalents per kilogram of protein, which is lower than that of conventional yellow pea protein isolate (5.3 kg CO2-equivalents/kg protein) and mild aqueous fractions (4.9 kg CO2-equivalents/kg protein)." } 603bb593b6a3c767252c90b93425f72e 2023-10-19 22:01:40.083 | INFO | llmsearch.database.crud:get_or_store_config:35 - Retrieved config id: 456b77b1-b0b9-428a-a1a7-591097804299 2023-10-19 22:01:40.098 | INFO | llmsearch.database.crud:create_response:78 - Saved response to the database. Response id: c6147862-3f98-41fd-8926-436c313f8c76

HyDE feature turned set to TRUE in config.yaml This is obviously not a proper response.

console output - ` 2023-10-19 21:40:07.395 | INFO | llmsearch.embeddings:get_embedding_model:30 - Embedding model config: type=<EmbeddingModelType.sentence_transformer: 'sentence_transformer'> model_name='BAAI/bge-large-en-v1.5' additional_kwargs={} 2023-10-19 21:40:08.813 | INFO | llmsearch.ranking:init:18 - Initializing Reranker... 2023-10-19 21:40:09.107 | INFO | llmsearch.splade:init:37 - Setting device to cuda:0 2023-10-19 21:40:09.886 | INFO | llmsearch.splade:load:111 - SPLADE: Got 1 labels. 2023-10-19 21:40:09.886 | INFO | llmsearch.splade:load:118 - Loaded sparse (SPLADE) embeddings from /home/hisma/rag/embeddings/splade/splade_embeddings.npz 2023-10-19 21:40:09.889 | INFO | llmsearch.utils:get_llm_bundle:78 - Initialized persistence db. 2023-10-19 21:40:09.889 | INFO | llmsearch.utils:get_hyde_chain:97 - Creating HyDE chain... [chain/start] [1:chain:LLMChain] Entering Chain run with input: { "question": "GHG footprint of pea protein" } [llm/start] [1:chain:LLMChain > 2:llm:CustomLlamaLangChainModel] Entering LLM run with input: { "prompts": [ "Write a short passage to answer the question: GHG footprint of pea protein" ] } vs. animal protein

Pea protein is often touted as a more sustainable alternative to animal protein due to its lower greenhouse gas (GHG) emissions. However, it's important to consider that the production process for pea protein can also have environmental impacts. For example, the cultivation of peas requires water and fertilizers, which can contribute to GHG emissions. Additionally, the energy required to process peas into protein powder or other forms also contributes to GHG emissions.

On the other hand, animal protein production is a significant contributor to GHG emissions due to factors such as livestock digestion and manure management. The methane produced by livestock during digestion is a potent greenhouse gas that contributes significantly to global warming. Furthermore, the use of land for grazing and feed crops also contributes to deforestation and habitat loss, which further exacerbates GHG emissions.

In conclusion, while pea protein may have a lower GHG footprint than animal protein in some cases, it's important to consider the full lifecycle emissions of both types of protein. The choice between the two should be based on a variety of factors, including personal health needs, environmental impact, and ethical considerations. llama_print_timings: load time = 472.70 ms llama_print_timings: sample time = 81.83 ms / 286 runs ( 0.29 ms per token, 3494.88 tokens per second) llama_print_timings: prompt eval time = 472.67 ms / 19 tokens ( 24.88 ms per token, 40.20 tokens per second) llama_print_timings: eval time = 17862.83 ms / 285 runs ( 62.68 ms per token, 15.95 tokens per second) llama_print_timings: total time = 18714.87 ms [llm/end] [1:chain:LLMChain > 2:llm:CustomLlamaLangChainModel] [18.72s] Exiting LLM run with output: { "generations": [ [ { "text": " vs. animal protein\n\nPea protein is often touted as a more sustainable alternative to animal protein due to its lower greenhouse gas (GHG) emissions. However, it's important to consider that the production process for pea protein can also have environmental impacts. For example, the cultivation of peas requires water and fertilizers, which can contribute to GHG emissions. Additionally, the energy required to process peas into protein powder or other forms also contributes to GHG emissions.\n\nOn the other hand, animal protein production is a significant contributor to GHG emissions due to factors such as livestock digestion and manure management. The methane produced by livestock during digestion is a potent greenhouse gas that contributes significantly to global warming. Furthermore, the use of land for grazing and feed crops also contributes to deforestation and habitat loss, which further exacerbates GHG emissions.\n\nIn conclusion, while pea protein may have a lower GHG footprint than animal protein in some cases, it's important to consider the full lifecycle emissions of both types of protein. The choice between the two should be based on a variety of factors, including personal health needs, environmental impact, and ethical considerations.", "generation_info": null } ] ], "llm_output": null, "run": null } [chain/end] [1:chain:LLMChain] [18.72s] Exiting Chain run with output: { "text": " vs. animal protein\n\nPea protein is often touted as a more sustainable alternative to animal protein due to its lower greenhouse gas (GHG) emissions. However, it's important to consider that the production process for pea protein can also have environmental impacts. For example, the cultivation of peas requires water and fertilizers, which can contribute to GHG emissions. Additionally, the energy required to process peas into protein powder or other forms also contributes to GHG emissions.\n\nOn the other hand, animal protein production is a significant contributor to GHG emissions due to factors such as livestock digestion and manure management. The methane produced by livestock during digestion is a potent greenhouse gas that contributes significantly to global warming. Furthermore, the use of land for grazing and feed crops also contributes to deforestation and habitat loss, which further exacerbates GHG emissions.\n\nIn conclusion, while pea protein may have a lower GHG footprint than animal protein in some cases, it's important to consider the full lifecycle emissions of both types of protein. The choice between the two should be based on a variety of factors, including personal health needs, environmental impact, and ethical considerations." } 2023-10-19 21:40:39.139 | INFO | llmsearch.process:get_hyde_response:137 - HYDE: got response: vs. animal protein

Pea protein is often touted as a more sustainable alternative to animal protein due to its lower greenhouse gas (GHG) emissions. However, it's important to consider that the production process for pea protein can also have environmental impacts. For example, the cultivation of peas requires water and fertilizers, which can contribute to GHG emissions. Additionally, the energy required to process peas into protein powder or other forms also contributes to GHG emissions.

On the other hand, animal protein production is a significant contributor to GHG emissions due to factors such as livestock digestion and manure management. The methane produced by livestock during digestion is a potent greenhouse gas that contributes significantly to global warming. Furthermore, the use of land for grazing and feed crops also contributes to deforestation and habitat loss, which further exacerbates GHG emissions.

In conclusion, while pea protein may have a lower GHG footprint than animal protein in some cases, it's important to consider the full lifecycle emissions of both types of protein. The choice between the two should be based on a variety of factors, including personal health needs, environmental impact, and ethical considerations. 2023-10-19 21:40:39.139 | INFO | llmsearch.ranking:get_relevant_documents:46 - Adding query prefix for retrieval: query: 2023-10-19 21:40:39.139 | INFO | llmsearch.splade:query:186 - SPLADE search will search over all documents of chunk size: 1024. Number of docs: 3045 [0.08197261 0.10693906 0.12125421 ... 0.11251494 0.15921329 0.09446842] 2023-10-19 21:40:39.174 | INFO | llmsearch.ranking:get_relevant_documents:61 - Stage 1: Got 15 documents. 2023-10-19 21:40:39.174 | INFO | llmsearch.ranking:get_relevant_documents:73 - Dense embeddings filter: None 2023-10-19 21:40:39.349 | INFO | llmsearch.ranking:rerank:23 - Reranking documents ... 2023-10-19 21:40:39.452 | INFO | llmsearch.ranking:rerank:32 - [2.5162177085876465, 1.3420692682266235, 1.2025434970855713, 1.0988667011260986, 1.034225583076477, 1.0273360013961792, 0.9255903959274292, 0.9013829231262207, 0.5510232448577881, 0.4137190878391266, 0.3995491862297058, 0.3035319447517395, 0.23208431899547577, 0.11294107884168625, 0.0032988302409648895, -0.24362006783485413, -0.644443690776825, -0.7955482006072998, -0.89765465259552, -1.0919475555419922, -1.115916132926941, -1.193403720855713, -1.2233260869979858, -1.3747527599334717, -1.483472228050232, -2.0981221199035645, -2.3056681156158447, -2.609255075454712] 2023-10-19 21:40:39.452 | INFO | llmsearch.ranking:get_relevant_documents:96 - Number of documents after stage 1 (sparse): 15 2023-10-19 21:40:39.452 | INFO | llmsearch.ranking:get_relevant_documents:99 - Number of documents after stage 2 (dense + sparse): 28 2023-10-19 21:40:39.452 | INFO | llmsearch.ranking:get_relevant_documents:102 - Re-ranker avg. scores for top 5 resuls, chunk size 1024: 1.44 [chain/start] [1:chain:StuffDocumentsChain] Entering Chain run with input: [inputs] [chain/start] [1:chain:StuffDocumentsChain > 2:chain:LLMChain] Entering Chain run with input: { "question": "GHG footprint of pea protein", "context": "passage: animal based protein needs an input of 10 kg plant proteins,\ndepending on meat production type (Reijnders and Soret, 2003).\nEnvironmental impacts in animal production are higher than in\nplant production due to direct emissions from animals, in partic-\nularly from enteric fermentation of ruminants and manure man-\nagement, and due to high feed consumption per kg meat produced.\nThere is an increasing pressure to find alternative sustainable\nprotein sources to meet the protein demands of the growing global\npopulation despite the limited agricultural land.\nTherefore, from an environmental point of view there is a need\nto consume less animal-based proteins and increase the intake of\nplant-based proteins. Also from a health perspective, a shift from\nanimal-based to plant-based protein consumption would be\nfavourable in Western countries where current red meat con-\nsumption levels are increasing the risk of cancers (WCRF, 2017).\nHowever, the increasing elderly population requires elevated pro-\n\npassage: but the protein yield and purity are also considerably lower. Overall, linking environmental impact to\nprotein purity and yield allows for a comprehensive selection of sustainable food ingredients.\n© 2021 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license\n(http://creativecommons.org/licenses/by/4.0/).\n1. Introduction\nReady-to-eat soups, vegan ice-cream, and meat replacers are\nprocessed food products require texturisers that are often protein-\nbased. These ingredients can be produced from animal products,\nsuch as whey protein isolate, or plants, such as pea protein isolate.\nEspecially plant proteins are of increasing importance due to\npressing climate issues, which require food producers to become\nmore sustainable (Aiking, 2011). Currently, proteins that originate\nfrom oil-, starch-, or protein-bearing crops are isolated by isoelec-\ntric precipitation, requiring chemicals such as caustic (NaOH) and\nhydrochloric acid (HCL) for extraction and precipitation (Passe\n\npassage: This study also demonstrates that the environmental impact\ndoes not differ much between meats produced with different feed\nprotein sources. Another conclusion is that the energy source (e.g.,\nelectricity production) plays an important role in the environmen-\ntal impact of the meals.\nIn order to achieve any environmental gain of replacing animal\nprotein with pea protein in meat products, more than 10% of the\nanimal protein needs to be replaced. It is significantly more envi-\nronmentally beneficial to provide a fully vegetarian meal than it\nis to replace 10% of the animal protein in a meal with vegetable\nprotein.\nThe results developed in this study highlight the importance of\nusing a life cycle approach, that is, it is not sufficient to only look at\nthe impact from agriculture when comparing vegetable and animal\nfoods.\n6.1. Future research\nFour different meals were analysed in this study. In order to\nbetter understand the environmental impact of meat and grain le-\ngumes meals further work should focus on:\n\npassage: most of the impact categories, while the pork produced with feed\nbased on peas has a higher impact than the one produced with\nsoya-based feed on eutrophication. The impact for the vegetarian\nmeal is lower, for most categories considerably lower, than the im-\npact of the other meals.\n5. Discussion\n5.1. Difference between protein sources\nThe study shows that vegetarian meals are associated with less\nenvironmental impact than meals with animal protein, wherein\neutrophying and acidifying emissions, as well as greenhouse gas\nemissions, are much lower; however, concerning energy use, veg-\netarian meals require about the same amount of energy as the\nmeals with animal protein, which is due to the greater energy de-\nmands of industrial pea burger processing (primarily from freezing\nthe product). The representative data used in this study for the pea\nburger processing energy requirements come from a relatively\nsmall-scale plant in Sweden, which means there is a potential for" } [llm/start] [1:chain:StuffDocumentsChain > 2:chain:LLMChain > 3:llm:CustomLlamaLangChainModel] Entering LLM run with input: { "prompts": [ "### Instruction:\nUse the following pieces of context to answer the question at the end. If answer isn't in the context, say that you don't know, don't try to make up an answer.\n\n### Context:\n---------------\npassage: animal based protein needs an input of 10 kg plant proteins,\ndepending on meat production type (Reijnders and Soret, 2003).\nEnvironmental impacts in animal production are higher than in\nplant production due to direct emissions from animals, in partic-\nularly from enteric fermentation of ruminants and manure man-\nagement, and due to high feed consumption per kg meat produced.\nThere is an increasing pressure to find alternative sustainable\nprotein sources to meet the protein demands of the growing global\npopulation despite the limited agricultural land.\nTherefore, from an environmental point of view there is a need\nto consume less animal-based proteins and increase the intake of\nplant-based proteins. Also from a health perspective, a shift from\nanimal-based to plant-based protein consumption would be\nfavourable in Western countries where current red meat con-\nsumption levels are increasing the risk of cancers (WCRF, 2017).\nHowever, the increasing elderly population requires elevated pro-\n\npassage: but the protein yield and purity are also considerably lower. Overall, linking environmental impact to\nprotein purity and yield allows for a comprehensive selection of sustainable food ingredients.\n© 2021 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license\n(http://creativecommons.org/licenses/by/4.0/).\n1. Introduction\nReady-to-eat soups, vegan ice-cream, and meat replacers are\nprocessed food products require texturisers that are often protein-\nbased. These ingredients can be produced from animal products,\nsuch as whey protein isolate, or plants, such as pea protein isolate.\nEspecially plant proteins are of increasing importance due to\npressing climate issues, which require food producers to become\nmore sustainable (Aiking, 2011). Currently, proteins that originate\nfrom oil-, starch-, or protein-bearing crops are isolated by isoelec-\ntric precipitation, requiring chemicals such as caustic (NaOH) and\nhydrochloric acid (HCL) for extraction and precipitation (Passe\n\npassage: This study also demonstrates that the environmental impact\ndoes not differ much between meats produced with different feed\nprotein sources. Another conclusion is that the energy source (e.g.,\nelectricity production) plays an important role in the environmen-\ntal impact of the meals.\nIn order to achieve any environmental gain of replacing animal\nprotein with pea protein in meat products, more than 10% of the\nanimal protein needs to be replaced. It is significantly more envi-\nronmentally beneficial to provide a fully vegetarian meal than it\nis to replace 10% of the animal protein in a meal with vegetable\nprotein.\nThe results developed in this study highlight the importance of\nusing a life cycle approach, that is, it is not sufficient to only look at\nthe impact from agriculture when comparing vegetable and animal\nfoods.\n6.1. Future research\nFour different meals were analysed in this study. In order to\nbetter understand the environmental impact of meat and grain le-\ngumes meals further work should focus on:\n\npassage: most of the impact categories, while the pork produced with feed\nbased on peas has a higher impact than the one produced with\nsoya-based feed on eutrophication. The impact for the vegetarian\nmeal is lower, for most categories considerably lower, than the im-\npact of the other meals.\n5. Discussion\n5.1. Difference between protein sources\nThe study shows that vegetarian meals are associated with less\nenvironmental impact than meals with animal protein, wherein\neutrophying and acidifying emissions, as well as greenhouse gas\nemissions, are much lower; however, concerning energy use, veg-\netarian meals require about the same amount of energy as the\nmeals with animal protein, which is due to the greater energy de-\nmands of industrial pea burger processing (primarily from freezing\nthe product). The representative data used in this study for the pea\nburger processing energy requirements come from a relatively\nsmall-scale plant in Sweden, which means there is a potential for\n---------------\n\n### Question: GHG footprint of pea protein\n### Response:" ] } Llama.generate: prefix-match hit The passage does not provide information on the specific greenhouse gas (GHG) footprint of pea protein. llama_print_timings: load time = 472.70 ms llama_print_timings: sample time = 7.18 ms / 25 runs ( 0.29 ms per token, 3483.84 tokens per second) llama_print_timings: prompt eval time = 2544.38 ms / 1094 tokens ( 2.33 ms per token, 429.97 tokens per second) llama_print_timings: eval time = 1297.13 ms / 24 runs ( 54.05 ms per token, 18.50 tokens per second) llama_print_timings: total time = 3874.70 ms [llm/end] [1:chain:StuffDocumentsChain > 2:chain:LLMChain > 3:llm:CustomLlamaLangChainModel] [3.88s] Exiting LLM run with output: { "generations": [ [ { "text": "The passage does not provide information on the specific greenhouse gas (GHG) footprint of pea protein.", "generation_info": null } ] ], "llm_output": null, "run": null } [chain/end] [1:chain:StuffDocumentsChain > 2:chain:LLMChain] [3.88s] Exiting Chain run with output: { "text": "The passage does not provide information on the specific greenhouse gas (GHG) footprint of pea protein." } [chain/end] [1:chain:StuffDocumentsChain] [3.88s] Exiting Chain run with output: { "output_text": "The passage does not provide information on the specific greenhouse gas (GHG) footprint of pea protein." } ad66febaca1e2925f1b0922184db199b 2023-10-19 21:40:43.334 | INFO | llmsearch.database.crud:get_or_store_config:35 - Retrieved config id: ee7baa05-769c-446f-ae8d-52b3a83e60b7 2023-10-19 21:40:43.348 | INFO | llmsearch.database.crud:create_response:78 - Saved response to the database. Response id: 4afb32bf-b023-4c13-b9d1-09178ff66586 `

So luckily I've managed to "fix" the problem by disabling hyde. It's a cool feature though, so would be good to understand why enabling hyde in the config file leads to deteriorated results.

[Hisma]

I can give you my theory as to what I think is going on, but it's of course only a theory. I see that the HyDE has its own separate prompt that it uses, provided in the util.py file.

When we see the "bad" response, we can see that it specifically states "the passage does not provide specific information..."

So it's almost as though the llm prompt and the hyde prompt are clashing somehow.
If I disable hyde then it doesn't get mixed up with the llm prompt and the results are good.

What do your thoughts?

[Hisma]

more spam. I compared the console outputs side-by-side. Here's one thing that stood out to me -

I will leave it at this for now. Hopefully I've provided enough info to help you find the root cause. Or perhaps this is just a problem on my end. But definitely we can see even the non-hyde results are being impacted by having hyde turned on.

[snexus]

Thank you for the detailed explanation - when HyDE is turned on, you can definitely expect different (in some cases worse) results - that's by design. HyDE doesn't work well in all scenarios - I think it is suitable for cases when a user only knows in general what they want to ask, but doesn't know domain-specific terms and language.

The way HyDE works - it is 2 step mechanism: 1) It generates an "artificial answer" to the original query, using LLM and a HyDE specific prompt, that is then used to search the documents, instead of the original query. As a result, the "most similar documents" might be completely different to the ones obtained without HyDE. 2) The context (most similar chunks) from step (1) is then submitted again to the LLM, together with the original question.

This is the diagram that explains it, from the HyDE paper (https://arxiv.org/pdf/2212.10496.pdf)

Regardless of what you have in the config, if you turn HyDE off in the UI - are you still getting different results compared to the previous version?

[Hisma]

no problem. Well one issue I see is it won't let me ask the same question twice if I turn hyde on/off. So if I ask a question with hyde on, don't like hte response, then try to ask the same question with hyde turned off in the UI, it just repeats hte same response I got with hyde. Also, as I mentinoed earlier, toggling hyde on/off causes the history to temporarily disappear. It comes back once a question is answered. But it does appear that toggling hyde off makes the answers more accurate again.
So looks like that as long as you can fix the bug where you can't submit the same question twice, and the UI issue where the history disappears when hyde is toggled, it should be okay. Because you'll certainly be in situations where you will want to a/b test an answer to a question with hyde on vs off.
Also I'll mention again that with your history feature, the most recent question is not displayed. You don't see it until it gets added to history.

Thanks for being so responsive.

[Hisma]

Also, thanks for explaining the use-cases for when hyde should be used. I'm not sure if it was explained to this level of detail in your readme docs (I don't think it was), but understanding it better now, I can know when to use it in the future!

[snexus]

Thanks again for the feedback, appreciate it!

Well one issue I see is it won't let me ask the same question twice if I turn hyde on/off. So if I ask a question with hyde on, don't like hte response, then try to ask the same question with hyde turned off in the UI, it just repeats hte same response I got with hyde.

Will fix it shortly.

Also, as I mentinoed earlier, toggling hyde on/off causes the history to temporarily disappear. It comes back once a question is answered.

This one is annoying, but quite tricky to fix since it is a design limitation of Streamlit - on any element update (a toggle box in this case), the entire page is refreshed, so chat_input element is being reset, and all the resulting output disappears.

Also I'll mention again that with your history feature, the most recent question is not displayed. You don't see it until it gets added to history.

Do you mean the current question being asked is not stored immediately in the history? I deliberately didn't put it in the history card, since it is an active question and the response is displayed in front of you. Do you think it should appear in the history "cards" straight away?

[Hisma]

no problem! To your last question, yes I think from UI it should show your latest question immediately in history. As I'm testing this apps capabilities I'm going through a list of questios, so being able to see the question I just asked in a quick glance is very useful. Also, you may go away for a while and come back and not remember what question you just asked. It creates a natural flow of question/answer, question/answer as you scroll through the chat window (I know this isn't quite a chatbot but close enough)

[snexus]

I've pushed an update to the feature branch - should have all the issues fixed besides the disappearing UI one. Please let me know if it works for you.

[Hisma]

the disappearing UI is only a minor annoyance compared to the other issues encountered. Can definitely live with that one.
I'll test later today and confirm.

[Hisma]

tested, works as expected. thank you!