Closed shuther closed 7 months ago
HanXudong
commented
7 months ago I guess there are some errors related to the Serper key.
Also, I just updated the code to handle different Serper response codes. Please pull the latest version, it should help you identify the bug.
shuther
commented
7 months ago ok, so using mistral I have this (not sure why I have some errors)
ERROR]2024-04-18 10:48:02,369 CheckWorthy.py:52: ====== Error: , the LLM response is: {
"MBZUAI is the first university in the world focused on artificial intelligence.": "Maybe (This statement might be verifiable with clear and specific references to support its factuality, such as reliable sources or official records.)"
}
[ERROR]2024-04-18 10:48:02,369 CheckWorthy.py:53: ====== Our input is: [[{'role': 'system', 'content': 'You are a helpful assistant designed to output JSON.'}, {'role': 'user', 'content': 'Your task is to evaluate each provided statement to determine if it presents information whose factuality can be objectively verified by humans, irrespective of the statement\'s current accuracy. Consider the following guidelines:\n1. Opinions versus Facts: Distinguish between opinions, which are subjective and not verifiable, and statements that assert factual information, even if broad or general. Focus on whether there\'s a factual claim that can be investigated.\n2. Clarity and Specificity: Statements must have clear and specific references to be verifiable (e.g., "he is a professor" is not verifiable without knowing who "he" is).\n3. Presence of Factual Information: Consider a statement verifiable if it includes factual elements that can be checked against evidence or reliable sources, even if the overall statement might be broad or incorrect.\nYour response should be in JSON format, with each statement as a key and either "Yes" or "No" as the value, along with a brief rationale for your decision.\n\nFor example, given these statements:\n1. Gary Smith is a distinguished professor of economics.\n2. He is a professor at MBZUAI.\n3. Obama is the president of the UK.\n\nThe expected output is:\n{\n "Gary Smith is a distinguished professor of economics.": "Yes (The statement contains verifiable factual information about Gary Smith\'s professional title and field.)",\n "He is a professor at MBZUAI.": "No (The statement cannot be verified due to the lack of clear reference to who \'he\' is.)",\n "Obama is the president of the UK.": "Yes (This statement contain verifiable information regarding the political leadership of a country.)"\n}\n\nFor these statements:\n1. MBZUAI is the first university in the world focused on artificial intelligence.\n\nThe output should be:\n'}]]
[ERROR]2024-04-18 10:48:21,235 CheckWorthy.py:52: ====== Error: , the LLM response is: {
"MBZUAI is the first university in the world focused on artificial intelligence.": "Maybe (This statement might be verifiable with clear and specific references to support its factuality, such as reliable sources or official records.)"
}
[ERROR]2024-04-18 10:48:21,235 CheckWorthy.py:53: ====== Our input is: [[{'role': 'system', 'content': 'You are a helpful assistant designed to output JSON.'}, {'role': 'user', 'content': 'Your task is to evaluate each provided statement to determine if it presents information whose factuality can be objectively verified by humans, irrespective of the statement\'s current accuracy. Consider the following guidelines:\n1. Opinions versus Facts: Distinguish between opinions, which are subjective and not verifiable, and statements that assert factual information, even if broad or general. Focus on whether there\'s a factual claim that can be investigated.\n2. Clarity and Specificity: Statements must have clear and specific references to be verifiable (e.g., "he is a professor" is not verifiable without knowing who "he" is).\n3. Presence of Factual Information: Consider a statement verifiable if it includes factual elements that can be checked against evidence or reliable sources, even if the overall statement might be broad or incorrect.\nYour response should be in JSON format, with each statement as a key and either "Yes" or "No" as the value, along with a brief rationale for your decision.\n\nFor example, given these statements:\n1. Gary Smith is a distinguished professor of economics.\n2. He is a professor at MBZUAI.\n3. Obama is the president of the UK.\n\nThe expected output is:\n{\n "Gary Smith is a distinguished professor of economics.": "Yes (The statement contains verifiable factual information about Gary Smith\'s professional title and field.)",\n "He is a professor at MBZUAI.": "No (The statement cannot be verified due to the lack of clear reference to who \'he\' is.)",\n "Obama is the president of the UK.": "Yes (This statement contain verifiable information regarding the political leadership of a country.)"\n}\n\nFor these statements:\n1. MBZUAI is the first university in the world focused on artificial intelligence.\n\nThe output should be:\n'}]]
[ERROR]2024-04-18 10:48:39,850 CheckWorthy.py:52: ====== Error: , the LLM response is: {
"MBZUAI is the first university in the world focused on artificial intelligence.": "Maybe (This statement might be verifiable with clear and specific references to support its factuality, such as reliable sources or official records.)"
}
[ERROR]2024-04-18 10:48:39,850 CheckWorthy.py:53: ====== Our input is: [[{'role': 'system', 'content': 'You are a helpful assistant designed to output JSON.'}, {'role': 'user', 'content': 'Your task is to evaluate each provided statement to determine if it presents information whose factuality can be objectively verified by humans, irrespective of the statement\'s current accuracy. Consider the following guidelines:\n1. Opinions versus Facts: Distinguish between opinions, which are subjective and not verifiable, and statements that assert factual information, even if broad or general. Focus on whether there\'s a factual claim that can be investigated.\n2. Clarity and Specificity: Statements must have clear and specific references to be verifiable (e.g., "he is a professor" is not verifiable without knowing who "he" is).\n3. Presence of Factual Information: Consider a statement verifiable if it includes factual elements that can be checked against evidence or reliable sources, even if the overall statement might be broad or incorrect.\nYour response should be in JSON format, with each statement as a key and either "Yes" or "No" as the value, along with a brief rationale for your decision.\n\nFor example, given these statements:\n1. Gary Smith is a distinguished professor of economics.\n2. He is a professor at MBZUAI.\n3. Obama is the president of the UK.\n\nThe expected output is:\n{\n "Gary Smith is a distinguished professor of economics.": "Yes (The statement contains verifiable factual information about Gary Smith\'s professional title and field.)",\n "He is a professor at MBZUAI.": "No (The statement cannot be verified due to the lack of clear reference to who \'he\' is.)",\n "Obama is the president of the UK.": "Yes (This statement contain verifiable information regarding the political leadership of a country.)"\n}\n\nFor these statements:\n1. MBZUAI is the first university in the world focused on artificial intelligence.\n\nThe output should be:\n'}]]
[INFO]2024-04-18 10:48:39,851 __init__.py:111: == State: Done! (Nothing to check.)
{
"0_response": "MBZUAI is the first AI university in the world",
"1_decompose": [
"MBZUAI is the first university in the world focused on artificial intelligence."
],
"2_checkworthy": [],
"2_checkworthy_pairwise": {
"MBZUAI is the first university in the world focused on artificial intelligence.": "Maybe (This statement might be verifiable with clear and specific references to support its factuality, such as reliable sources or official records.)"
},
"3_query_generator": {},
"4_evidence_retrieve": {},
"5_claim_verify": {}
}
with wizardlm2 I have this; not sure I understand why the API key is a problem as it is self hosted: poetry run python -m factcheck --modal string --input "MBZUAI is the first AI university in the world" --client local_openai --model wizardlm2 --prompt factcheck/config/sample_prompt.yaml
== Init decompose_model with model: wizardlm2
[INFO]2024-04-18 11:00:07,223 __init__.py:53: == Use specified client: local_openai
== Init checkworthy_model with model: wizardlm2
[INFO]2024-04-18 11:00:07,223 __init__.py:53: == Use specified client: local_openai
== Init query_generator_model with model: wizardlm2
[INFO]2024-04-18 11:00:07,223 __init__.py:53: == Use specified client: local_openai
== Init evidence_retrieval_model with model: wizardlm2
[INFO]2024-04-18 11:00:07,223 __init__.py:53: == Use specified client: local_openai
== Init claim_verify_model with model: wizardlm2
[INFO]2024-04-18 11:00:07,223 __init__.py:53: == Use specified client: local_openai
[INFO]2024-04-18 11:00:07,223 __init__.py:67: ===Sub-modules Init Finished===
[INFO]2024-04-18 11:00:07,223 multimodal.py:89: == Processing: Modal: string, Input: MBZUAI is the first AI university in the world
[INFO]2024-04-18 11:00:07,223 multimodal.py:103: == Processed: Modal: string, Input: MBZUAI is the first AI university in the world
[INFO]2024-04-18 11:01:17,452 __init__.py:78: == response claims 0: MBZUAI was established as the first artificial intelligence university globally.
[INFO]2024-04-18 11:01:17,452 __init__.py:78: == response claims 1: The Mohammed Bin Zayed University of Artificial Intelligence (MBZUAI) was founded.
[INFO]2024-04-18 11:03:55,539 __init__.py:86: == Check-worthy claims 0: The Mohammed Bin Zayed University of Artificial Intelligence (MBZUAI) was founded.
[INFO]2024-04-18 11:05:50,913 __init__.py:117: == Claim: The Mohammed Bin Zayed University of Artificial Intelligence (MBZUAI) was founded. --- Queries: ['The Mohammed Bin Zayed University of Artificial Intelligence (MBZUAI) was founded.', 'Who founded the Mohammed Bin Zayed University of Artificial Intelligence?', 'When was the Mohammed Bin Zayed University of Artificial Intelligence established?']
[INFO]2024-04-18 11:05:50,916 SerperEvidenceRetrieve.py:30: Collecting evidences ...
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/__main__.py", line 45, in <module>
check(args)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/__main__.py", line 30, in check
res = factcheck.check_response(content)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/__init__.py", line 122, in check_response
claim_evidence_dict = self.evidence_crawler.retrieve_evidence(claim_query_dict=claim_query_dict)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/core/Retriever/SerperEvidenceRetrieve.py", line 33, in retrieve_evidence
evidence_list = self._retrieve_evidence_4_all_claim(
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/core/Retriever/SerperEvidenceRetrieve.py", line 62, in _retrieve_evidence_4_all_claim
serper_response = self._request_serper_api(query_list)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/core/Retriever/SerperEvidenceRetrieve.py", line 193, in _request_serper_api
raise Exception("Failed to authenticate. Check your API key.")
Exception: Failed to authenticate. Check your API key.ok for serper, I will set a key and try again today
shuther
commented
7 months ago I am not sure why my serper API key is not accepted. is there a way to debug it further:
[INFO]2024-04-18 18:41:18,230 __init__.py:117: == Claim: The Mohammed Bin Zayed University of Artificial Intelligence (MBZUAI) was founded. --- Queries: ['The Mohammed Bin Zayed University of Artificial Intelligence (MBZUAI) was founded.', 'Who founded the Mohammed Bin Zayed University of Artificial Intelligence?', 'When was the Mohammed Bin Zayed University of Artificial Intelligence established?']
[INFO]2024-04-18 18:41:18,245 SerperEvidenceRetrieve.py:30: Collecting evidences ...
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/__main__.py", line 45, in <module>
check(args)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/__main__.py", line 30, in check
res = factcheck.check_response(content)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/__init__.py", line 122, in check_response
claim_evidence_dict = self.evidence_crawler.retrieve_evidence(claim_query_dict=claim_query_dict)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/core/Retriever/SerperEvidenceRetrieve.py", line 33, in retrieve_evidence
evidence_list = self._retrieve_evidence_4_all_claim(
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/core/Retriever/SerperEvidenceRetrieve.py", line 62, in _retrieve_evidence_4_all_claim
serper_response = self._request_serper_api(query_list)
File "/home/shuther/Documents/Projects/OpenFactVerification/factcheck/core/Retriever/SerperEvidenceRetrieve.py", line 193, in _request_serper_api
raise Exception("Failed to authenticate. Check your API key.")
Exception: Failed to authenticate. Check your API key.Hi, to assist with debugging Serper, I have added a sample function. Please try this
python -m factcheck.core.Retriever.SerperEvidenceRetrieve --serper_api_key YOUR_SERPER_KEYIf your Serper key is correct, you shall be able to see results. If still getting the same error, please try to get a new key from https://serper.dev/
shuther
commented
7 months ago it was my fault, I made a typo :-(
not sure about the origin of the problem. with the model mistral is worked but it didn't go that far.