Open tom-doerr opened 3 months ago
MIPRO and assertions not compatible currently unfortunately
Ah okay. Would it make sense to throw that as the error message? I spent quite a bit of time trying to solve this myself
Feel free to add it @tom-doerr ! Assertions are currently compatible with the FewShot optimizers, and we'll eventually get them integrated with the new Signature optimizers coming out.
MIPRO and assertions not compatible currently unfortunately
I encountered the same error when using BootstrapFewShotWithRandomSearch. Does BootstrapFewShotWithRandomSearch also not support assertions?
Hey, I'm also seeing this with BootstrapFewShotWithRandomSearch and Suggestion.
Here's the error:
2024-09-09 01:34:17,991 - dspy.primitives.assertions - INFO - [2m2024-09-09T05:34:17.990976Z[0m [[32m[1minfo [0m] [1mSuggestionFailed: The quizbowl question does not contain the exact, specified number of sentences/clues. Please rewrite it such that it has exactly the specified number (num_clues).[0m [[0m[1m[34mdspy.primitives.assertions[0m][0m [36mfilename[0m=[35massertions.py[0m [36mlineno[0m=[35m108[0m
Traceback (most recent call last):
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 220, in wrapper
result = func(*args, **kwargs)
File "/some/dir/some/dir/question_generation/pyramodelQBGeneratorDSPy.py", line 234, in forward
dspy.Suggest(
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 74, in __init__
self.__call__()
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 112, in __call__
raise DSPySuggestionError(
dspy.primitives.assertions.DSPySuggestionError: The quizbowl question contains facts that are not grounded in the context of the Wikipedia sentences it was given. Please revise for accuracy.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/some/dir/some/dir/question_generation/pyramodelDSPyTrainer.py", line 114, in <module>
main(args)
File "/some/dir/some/dir/question_generation/pyramodelDSPyTrainer.py", line 68, in main
optimized_program = teleprompter.compile(student=pyraGenerator, teacher=pyraGenerator, trainset=train_dataset, valset=val_dataset)
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/teleprompt/random_search.py", line 123, in compile
score, subscores = evaluate(program2, return_all_scores=True)
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/evaluate/evaluate.py", line 193, in __call__
reordered_devset, ncorrect, ntotal = self._execute_multi_thread(
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/evaluate/evaluate.py", line 110, in _execute_multi_thread
example_idx, example, prediction, score = future.result()
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/evaluate/evaluate.py", line 103, in cancellable_wrapped_program
return wrapped_program(idx, arg)
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/evaluate/evaluate.py", line 160, in wrapped_program
prediction = program(**example.inputs())
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/program.py", line 26, in __call__
return self.forward(*args, **kwargs)
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 294, in forward
return wrapped_forward(*args, **kwargs)
File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 260, in wrapper
output_fields = error_state[0].new_signature.output_fields
AttributeError: 'Predict' object has no attribute 'new_signature'. Did you mean: 'signature'?
CRITICAL: Exiting due to uncaught exception AttributeError: 'Predict' object has no attribute 'new_signature'
Here's the relevant code:
training script
pyraGenerator = assert_transform_module(PyramodelQbQuestionGeneratorDSPy().map_named_predictors(Retry), backtrack_handler)
pyra_df = pd.read_csv(filepath_or_buffer=DATA_DIR / f"dspy_training/pyra_questions.csv")
pyra_dataset = dl.from_pandas(
pyra_df,
fields=("question", "clues", "wiki_sents", "num_clues", "wiki_page_title", "max_tokens"),
input_keys=("clues", "wiki_sents", "num_clues", "wiki_page_title", "max_tokens")
)
splits = dl.train_test_split(pyra_dataset, train_size=0.5)
train_dataset = splits['train']
val_dataset = splits['test']
config = dict(max_bootstrapped_demos=16, max_labeled_demos=16, num_candidate_programs=16, num_threads=16)
teleprompter = BootstrapFewShotWithRandomSearch(metric=composite_qb_metric, **config)
optimized_program = teleprompter.compile(student=pyraGenerator, teacher=pyraGenerator, trainset=train_dataset, valset=val_dataset)
optimized_program.save(DATA_DIR / f"dspy_training/checkpoint.json", save_field_meta=True)
generator
class AssessQBQuestion(dspy.Signature):
"""Assess the quality of a generated quizbowl question along the specified dimension."""
assessment_question = dspy.InputField(desc="Question used to assess the quality of given quizbowl question.")
assessed_quizbowl_question = dspy.InputField(desc="Quizbowl question being assessed")
wiki_page_title = dspy.InputField(desc="Name of relevant Wikipedia page about the answer")
num_clues = dspy.InputField(desc="Number of clues in the quizbowl question. Typically, 3-8. Ignore if N/A")
max_tokens = dspy.InputField(desc="Number of allowed tokens. Ignore if N/A.")
context = dspy.InputField(desc="Ignore if N/A.")
assessment_answer: str = dspy.OutputField(desc="Yes or No")
IS_FAITHFUL = \
("Is the assessed quizbowl question grounded in the context? Say no if "
"it includes significant facts not in the context.")
def is_faithful(gold, pred, trace=None) -> bool:
return is_assessment_yes(
dspy.Predict(AssessQBQuestion)(
assessed_quizbowl_question=pred.generated_question,
assessment_question=IS_FAITHFUL,
wiki_page_title=pred.wiki_page_title,
num_clues=NA, max_tokens=NA,
context=pred.wiki_sents
)
)
class CluesToQBQuestion(dspy.Signature):
"""Write the list of Wikipedia sentences into an ACF-style quizbowl question."""
wiki_page_title: str = dspy.InputField(desc="The title of a Wikipedia page about the answer of the quizbowl question.")
num_clues: str = dspy.InputField(desc="The number of clues (sentences) to be in the resulting quizbowl question.")
max_tokens: str = dspy.InputField(desc="The maximum allowed tokens.")
clues: str = dspy.InputField(desc="A list of quizbowl clues.")
quizbowl_question: str = dspy.OutputField(desc="The generated quizbowl question")
generator (forward)
def forward(
self,
wiki_page_title: str,
wiki_sents: List[str],
num_clues: str,
max_tokens: str,
clues: str=[] # Optional argument because it will be provided during training but not inference
) -> dspy.Prediction:
if type(num_clues) == int: num_clues = str(num_clues) # Typing with DSPy is not fun
if type(max_tokens) == int: max_tokens = str(max_tokens)
if not clues:
clues = []
for i, sent in enumerate(wiki_sents):
clue = \
self.clueMasker.mask_wiki_sent(
wiki_page=wiki_page_title,
wiki_sent=sent,
clue_pos=(1+i)/num_clues)
clues.append(clue)
keywords = guesser.get_topk_keywords(
label=wiki_page_title, wiki_sents=wiki_sents, k=10)
self.mask_keywords(wiki_page_title=wiki_page_title, wiki_sents=clues, mask_first_k=2, keywords=keywords)
generated_question = self.finishing_touches(wiki_page_title, num_clues, max_tokens, clues)
# The keywords property is used by a metric to make sure keywords have been removed
pred = dspy.Prediction(
generated_question=generated_question,
wiki_sents=wiki_sents,
num_clues=num_clues,
max_tokens=str(int(max_tokens) + random.randint(-20, 20)),
wiki_page_title=wiki_page_title,
page=wiki_page_title)
dspy.Suggest(
is_faithful(None, pred),
msg=("The quizbowl question contains facts that are not grounded "
"in the context of the Wikipedia sentences it was given. "
"Please revise for accuracy."),
# target_module=CluesToQBQuestion,
is_metric=True
)
I commented out # target_module=CluesToQBQuestion
because I'm getting this issue
@tom-doerr Could you please elaborate on how you fixed this since it looks like you've fixed this problem (issue)?
I don't quite understand what is meant by "updated the signature class (target_module
)".
Suggest
takes the attribute target_module
:
dspy.Suggest(is_assessment_yes(contact_person_assessment.assessment_answer), "The text contains names and titles not in the context. Please revise for accuracy.", target_module=GenerateMail)
I believe I renamed GenerateMail
to something else. If I remember correctly I just updated the class name.
Interesting thank you @tom-doerr ! I'll take a look and update this issue if I find anything.
I'm getting this error when running MIPRO:
This is how I run MIPRO:
More output: