'Predict' object has no attribute 'new_signature'.

[tom-doerr]

I'm getting this error when running MIPRO:

AttributeError: 'Predict' object has no attribute 'new_signature'. Did you mean: 'signature'?

This is how I run MIPRO:

            teleprompter = dspy.teleprompt.MIPRO(metric = great_tweet_metric, verbose=True)
            kwargs = dict(num_threads=16, display_progress=True, display_table=0)
            tweet_generator_compiled = teleprompter.compile(tweet_generator, trainset=trainset,  num_trials=2, max_bootstrapped_demos=4, max_labeled_demos=5, eval_kwargs=kwargs, requires_permission_to_run=False)

More output:

  0%|                                             | 0/50 [00:00<?, ?it/s]2024-06-06T01:00:54.964938Z [error    ] Failed to run or to evaluate example Example({'reference_text': '---\nsidebar_position: 2\n---\n\n# retrieve.ChromadbRM\n\n### Constructor\n\nInitialize an instance of the `ChromadbRM` class, with the option to use OpenAI\'s embeddings or any alternative supported by chromadb, as detailed in the official [chromadb embeddings documentation](https://docs.trychroma.com/embeddings).\n\n```python\nChromadbRM(\n    collection_name: str,\n    persist_directory: str,\n    embedding_function: Optional[EmbeddingFunction[Embeddable]] = OpenAIEmbeddingFunction(),\n    k: int = 7,\n)\n```\n\n**Parameters:**\n- `collection_name` (_str_): The name of the chromadb collection.\n- `persist_directory` (_str_): Path to the directory where chromadb data is persisted.\n- `embedding_function` (_Optional[EmbeddingFunction[Embeddable]]_, _optional_): The function used for embedding documents and queries. Defaults to `DefaultEmbeddingFunction()` if not specified.\n- `k` (_int_, _optional_): The number of top passages to retrieve. Defaults to 7.\n\n### Methods\n\n#### `forward(self, query_or_queries: Union[str, List[str]], k: Optional[int] = None) -> dspy.Prediction`\n\nSearch the chromadb collection for the top `k` passages matching the given query or queries, using embeddings generated via the specified `embedding_function`.\n\n**Parameters:**\n- `query_or_queries` (_Union[str, List[str]]_): The query or list of queries to search for.\n- `k` (_Optional[int]_, _optional_): The number of results to retrieve. If not specified, defaults to the value set during initialization.\n\n**Returns:**\n- `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with schema `[{"id": str, "score": float, "long_text": str, "metadatas": dict }]`\n\n### Quickstart with OpenAI Embeddings\n\nChromadbRM have the flexibility from a variety of embedding functions as outlined in the [chromadb embeddings documentation](https://docs.trychroma.com/embeddings). While different options are available, this example demonstrates how to utilize OpenAI embeddings specifically.\n\n```python\nfrom dspy.retrieve.chromadb_rm import ChromadbRM\nimport os\nimport openai\nfrom chromadb.utils.embedding_functions import OpenAIEmbeddingFunction\n\nembedding_function = OpenAIEmbeddingFunction(\n    api_key=os.environ.get(\'OPENAI_API_KEY\'),\n    model_name="text-embedding-ada-002"\n)\n\nretriever_model = ChromadbRM(\n    \'your_collection_name\',\n    \'/path/to/your/db\',\n    embedding_function=embedding_function,\n    k=5\n)\n\nresults = retriever_model("Explore the significance of quantum computing", k=5)\n\nfor result in results:\n    print("Document:", result.long_text, "\\n")\n```\n'}) (input_keys={'reference_text'}) with <function great_tweet_metric at 0x7f2d108eb010> due to 'Predict' object has no attribute 'new_signature'. [dspy.teleprompt.bootstrap] filename=bootstrap.py lineno=204
2024-06-06T01:00:54.969715Z [error    ] Failed to run or to evaluate example Example({'reference_text': '---\nsidebar_position: \n---\n\n# dspy.Snowflake\n\n### Usage\n\n```python\nimport dspy\nimport os\n\nconnection_parameters = {\n\n    "account": os.getenv(\'SNOWFLAKE_ACCOUNT\'),\n    "user": os.getenv(\'SNOWFLAKE_USER\'),\n    "password": os.getenv(\'SNOWFLAKE_PASSWORD\'),\n    "role": os.getenv(\'SNOWFLAKE_ROLE\'),\n    "warehouse": os.getenv(\'SNOWFLAKE_WAREHOUSE\'),\n    "database": os.getenv(\'SNOWFLAKE_DATABASE\'),\n    "schema": os.getenv(\'SNOWFLAKE_SCHEMA\')}\n\nlm = dspy.Snowflake(model="mixtral-8x7b",credentials=connection_parameters)\n```\n\n### Constructor\n\nThe constructor inherits from the base class `LM` and verifies the `credentials` for using Snowflake API.\n\n```python\nclass Snowflake(LM):\n    def __init__(\n        self, \n        model,\n        credentials,\n        **kwargs):\n```\n\n**Parameters:**\n- `model` (_str_): model hosted by [Snowflake Cortex](https://docs.snowflake.com/en/user-guide/snowflake-cortex/llm-functions#availability).\n- `credentials`  (_dict_): connection parameters required to initialize a [snowflake snowpark session](https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/latest/api/snowflake.snowpark.Session)\n\n### Methods\n\nRefer to [`dspy.Snowflake`](https://dspy-docs.vercel.app/api/language_model_clients/Snowflake) documentation.\n'}) (input_keys={'reference_text'}) with <function great_tweet_metric at 0x7f2d108eb010> due to 'Predict' object has no attribute 'new_signature'. [dspy.teleprompt.bootstrap] filename=bootstrap.py lineno=2042024-06-06T01:00:54.990462Z [error    ] Failed to run or to evaluate example Example({'reference_text': '---\nsidebar_position: 3\n---\n\n# teleprompt.Ensemble\n\n### Constructor\n\nThe constructor initializes the `Ensemble` class and sets up its attributes. This teleprompter is designed to create ensembled versions of multiple programs, reducing various outputs from different programs into a single output.\n\n```python\nclass Ensemble(Teleprompter):\n    def __init__(self, *, reduce_fn=None, size=None, deterministic=False):\n```\n\n**Parameters:**\n- `reduce_fn` (_callable_, _optional_): Function used to reduce multiple outputs from different programs into a single output. A common choice is `dspy.majority`. Defaults to `None`.\n- `size` (_int_, _optional_): Number of programs to randomly select for ensembling. If not specified, all programs will be used. Defaults to `None`.\n- `deterministic` (_bool_, _optional_): Specifies whether ensemble should operate deterministically. Currently, setting this to `True` will raise an error as this feature is pending implementation. Defaults to `False`.\n\n### Method\n\n#### `compile(self, programs)`\n\nThis method compiles an ensemble of programs into a single program that when run, can either randomly sample a subset of the given programs to produce outputs or use all of them. The multiple outputs can then be reduced into a single output using the `reduce_fn`.\n\n**Parameters:**\n- `programs` (_list_): List of programs to be ensembled.\n\n**Returns:**\n- `EnsembledProgram` (_Module_): An ensembled version of the input programs.\n\n### Example\n\n```python\nimport dspy\nfrom dspy.teleprompt import Ensemble\n\n# Assume a list of programs\nprograms = [program1, program2, program3, ...]\n\n# Define Ensemble teleprompter\nteleprompter = Ensemble(reduce_fn=dspy.majority, size=2)\n\n# Compile to get the EnsembledProgram\nensembled_program = teleprompter.compile(programs)\n```'}) (input_keys={'reference_text'}) with <function great_tweet_metric at 0x7f2d108eb010> due to 'Predict' object has no attribute 'new_signature'. [dspy.teleprompt.bootstrap] filename=bootstrap.py lineno=204
2024-06-06T01:00:54.995006Z [error    ] Failed to run or to evaluate example Example({'reference_text': '---\nsidebar_position: 5\n---\n\n# dspy.HFClientVLLM\n\n### Usage\n\n```python\nlm = dspy.HFClientVLLM(model="meta-llama/Llama-2-7b-hf", port=8080, url="http://localhost")\n```\n\n### Prerequisites\n\nRefer to the [vLLM Server](https://dspy-docs.vercel.app/api/language_model_clients/HFClientVLLM) section of the `Using Local Models` documentation.\n\n### Constructor\n\nRefer to [`dspy.TGI`](https://dspy-docs.vercel.app/api/language_model_clients/TGI) documentation. Replace with `HFClientVLLM`.\n\n### Methods\n\nRefer to [`dspy.OpenAI`](https://dspy-docs.vercel.app/api/language_model_clients/OpenAI) documentation.'}) (input_keys={'reference_text'}) with <function great_tweet_metric at 0x7f2d108eb010> due to 'Predict' object has no attribute 'new_signature'. [dspy.teleprompt.bootstrap] filename=bootstrap.py lineno=204
 14%|█████                               | 7/50 [00:00<00:00, 147.64it/s]
Traceback (most recent call last):
  File "/home/tom/dspy/dspy/primitives/assertions.py", line 220, in wrapper
    result = func(*args, **kwargs)
  File "/home/tom/tweet_generation/./main.py", line 127, in forward
    run_all_checks(tweet_text)
  File "/home/tom/tweet_generation/./main.py", line 110, in run_all_checks
    suggest_check(generated_text, "Does the assessed tweet sound natural? Say no if it sounds like an ad.", "The tweet does not sound natural. Please revise for quality.")
  File "/home/tom/tweet_generation/./main.py", line 100, in suggest_check
    dspy.Suggest(assessment_bool, suggestion)
  File "/home/tom/dspy/dspy/primitives/assertions.py", line 74, in __init__
    self.__call__()
  File "/home/tom/dspy/dspy/primitives/assertions.py", line 112, in __call__
    raise DSPySuggestionError(
dspy.primitives.assertions.DSPySuggestionError: The tweet does not sound natural. Please revise for quality.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tom/tweet_generation/./main.py", line 189, in <module>
    tweet_generator_compiled = teleprompter.compile(tweet_generator, trainset=trainset,  num_trials=2, max_bootstrapped_demos=4, max_labeled_demos=5, eval_kwargs=kwargs, requires_permission_to_run=False)
  File "/home/tom/dspy/dspy/teleprompt/mipro_optimizer.py", line 452, in compile
    candidate_program = tp.compile(student=module.deepcopy(), trainset=shuffled_trainset)
  File "/home/tom/dspy/dspy/teleprompt/bootstrap.py", line 84, in compile
    self._bootstrap()
  File "/home/tom/dspy/dspy/teleprompt/bootstrap.py", line 147, in _bootstrap
    success = self._bootstrap_one_example(example, round_idx)
  File "/home/tom/dspy/dspy/teleprompt/bootstrap.py", line 203, in _bootstrap_one_example
    raise e
  File "/home/tom/dspy/dspy/teleprompt/bootstrap.py", line 183, in _bootstrap_one_example
    prediction = teacher(**example.inputs())
  File "/home/tom/dspy/dspy/primitives/program.py", line 26, in __call__
    return self.forward(*args, **kwargs)
  File "/home/tom/dspy/dspy/primitives/assertions.py", line 294, in forward
    return wrapped_forward(*args, **kwargs)
  File "/home/tom/dspy/dspy/primitives/assertions.py", line 260, in wrapper
    output_fields = error_state[0].new_signature.output_fields
AttributeError: 'Predict' object has no attribute 'new_signature'. Did you mean: 'signature'?

[okhat]

MIPRO and assertions not compatible currently unfortunately

[tom-doerr]

Ah okay. Would it make sense to throw that as the error message? I spent quite a bit of time trying to solve this myself

[arnavsinghvi11]

Feel free to add it @tom-doerr ! Assertions are currently compatible with the FewShot optimizers, and we'll eventually get them integrated with the new Signature optimizers coming out.

[LMWL-HL]

MIPRO and assertions not compatible currently unfortunately

I encountered the same error when using BootstrapFewShotWithRandomSearch. Does BootstrapFewShotWithRandomSearch also not support assertions?

[YichiRockyZhang]

Hey, I'm also seeing this with BootstrapFewShotWithRandomSearch and Suggestion.

Here's the error:

2024-09-09 01:34:17,991 - dspy.primitives.assertions - INFO - 2024-09-09T05:34:17.990976Z [info     ] SuggestionFailed: The quizbowl question does not contain the exact, specified number of sentences/clues. Please rewrite it such that it has exactly the specified number (num_clues). [dspy.primitives.assertions] filename=assertions.py lineno=108
Traceback (most recent call last):
  File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 220, in wrapper
    result = func(*args, **kwargs)
  File "/some/dir/some/dir/question_generation/pyramodelQBGeneratorDSPy.py", line 234, in forward
    dspy.Suggest(
  File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 74, in __init__
    self.__call__()
  File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 112, in __call__
    raise DSPySuggestionError(
dspy.primitives.assertions.DSPySuggestionError: The quizbowl question contains facts that are not grounded in the context of the Wikipedia sentences it was given. Please revise for accuracy.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/some/dir/some/dir/question_generation/pyramodelDSPyTrainer.py", line 114, in <module>
    main(args)
  File "/some/dir/some/dir/question_generation/pyramodelDSPyTrainer.py", line 68, in main
    optimized_program = teleprompter.compile(student=pyraGenerator, teacher=pyraGenerator, trainset=train_dataset, valset=val_dataset)
  File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/teleprompt/random_search.py", line 123, in compile
    score, subscores = evaluate(program2, return_all_scores=True)
  File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/evaluate/evaluate.py", line 193, in __call__
    reordered_devset, ncorrect, ntotal = self._execute_multi_thread(
  File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/evaluate/evaluate.py", line 110, in _execute_multi_thread
    example_idx, example, prediction, score = future.result()
  File "/some/dir/miniconda3/envs/qa5/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/some/dir/miniconda3/envs/qa5/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/some/dir/miniconda3/envs/qa5/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/evaluate/evaluate.py", line 103, in cancellable_wrapped_program
    return wrapped_program(idx, arg)
  File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/evaluate/evaluate.py", line 160, in wrapped_program
    prediction = program(**example.inputs())
  File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/program.py", line 26, in __call__
    return self.forward(*args, **kwargs)
  File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 294, in forward
    return wrapped_forward(*args, **kwargs)
  File "/some/dir/miniconda3/envs/qa5/lib/python3.10/site-packages/dspy/primitives/assertions.py", line 260, in wrapper
    output_fields = error_state[0].new_signature.output_fields
AttributeError: 'Predict' object has no attribute 'new_signature'. Did you mean: 'signature'?
CRITICAL: Exiting due to uncaught exception AttributeError: 'Predict' object has no attribute 'new_signature'

Here's the relevant code:

training script

    pyraGenerator = assert_transform_module(PyramodelQbQuestionGeneratorDSPy().map_named_predictors(Retry), backtrack_handler) 

    pyra_df = pd.read_csv(filepath_or_buffer=DATA_DIR / f"dspy_training/pyra_questions.csv")

    pyra_dataset = dl.from_pandas(
        pyra_df,
        fields=("question", "clues", "wiki_sents", "num_clues", "wiki_page_title", "max_tokens"),
        input_keys=("clues", "wiki_sents", "num_clues", "wiki_page_title", "max_tokens")
    )

    splits = dl.train_test_split(pyra_dataset, train_size=0.5)
    train_dataset = splits['train']
    val_dataset = splits['test']

    config = dict(max_bootstrapped_demos=16, max_labeled_demos=16, num_candidate_programs=16, num_threads=16)
    teleprompter = BootstrapFewShotWithRandomSearch(metric=composite_qb_metric, **config)
    optimized_program = teleprompter.compile(student=pyraGenerator, teacher=pyraGenerator, trainset=train_dataset, valset=val_dataset)
    optimized_program.save(DATA_DIR / f"dspy_training/checkpoint.json", save_field_meta=True)

generator

class AssessQBQuestion(dspy.Signature):
    """Assess the quality of a generated quizbowl question along the specified dimension."""
    assessment_question = dspy.InputField(desc="Question used to assess the quality of given quizbowl question.")
    assessed_quizbowl_question = dspy.InputField(desc="Quizbowl question being assessed")
    wiki_page_title = dspy.InputField(desc="Name of relevant Wikipedia page about the answer")
    num_clues = dspy.InputField(desc="Number of clues in the quizbowl question. Typically, 3-8. Ignore if N/A")
    max_tokens = dspy.InputField(desc="Number of allowed tokens. Ignore if N/A.")
    context = dspy.InputField(desc="Ignore if N/A.")

    assessment_answer: str = dspy.OutputField(desc="Yes or No")

IS_FAITHFUL = \
("Is the assessed quizbowl question grounded in the context? Say no if "
 "it includes significant facts not in the context.")
def is_faithful(gold, pred, trace=None) -> bool:
    return is_assessment_yes(
        dspy.Predict(AssessQBQuestion)(
            assessed_quizbowl_question=pred.generated_question,
            assessment_question=IS_FAITHFUL,
            wiki_page_title=pred.wiki_page_title,
            num_clues=NA, max_tokens=NA,
            context=pred.wiki_sents
        )
    )

class CluesToQBQuestion(dspy.Signature):
    """Write the list of Wikipedia sentences into an ACF-style quizbowl question."""
    wiki_page_title: str = dspy.InputField(desc="The title of a Wikipedia page about the answer of the quizbowl question.")
    num_clues: str = dspy.InputField(desc="The number of clues (sentences) to be in the resulting quizbowl question.")
    max_tokens: str = dspy.InputField(desc="The maximum allowed tokens.")
    clues: str = dspy.InputField(desc="A list of quizbowl clues.")

    quizbowl_question: str = dspy.OutputField(desc="The generated quizbowl question")

generator (forward)

    def forward(
        self,
        wiki_page_title: str,
        wiki_sents: List[str],
        num_clues: str,
        max_tokens: str,
        clues: str=[] # Optional argument because it will be provided during training but not inference
        ) -> dspy.Prediction:

        if type(num_clues) == int: num_clues = str(num_clues) # Typing with DSPy is not fun
        if type(max_tokens) == int: max_tokens = str(max_tokens)

        if not clues:
            clues = []
            for i, sent in enumerate(wiki_sents):
                clue = \
                    self.clueMasker.mask_wiki_sent(
                        wiki_page=wiki_page_title,
                        wiki_sent=sent,
                        clue_pos=(1+i)/num_clues)
                clues.append(clue)

        keywords = guesser.get_topk_keywords(
            label=wiki_page_title, wiki_sents=wiki_sents, k=10)

        self.mask_keywords(wiki_page_title=wiki_page_title, wiki_sents=clues, mask_first_k=2, keywords=keywords)
        generated_question = self.finishing_touches(wiki_page_title, num_clues, max_tokens, clues)

        # The keywords property is used by a metric to make sure keywords have been removed
        pred = dspy.Prediction(
            generated_question=generated_question,
            wiki_sents=wiki_sents,
            num_clues=num_clues,
            max_tokens=str(int(max_tokens) + random.randint(-20, 20)),
            wiki_page_title=wiki_page_title,
            page=wiki_page_title)

        dspy.Suggest(
            is_faithful(None, pred),
            msg=("The quizbowl question contains facts that are not grounded "
                    "in the context of the Wikipedia sentences it was given. "
                    "Please revise for accuracy."),
            # target_module=CluesToQBQuestion,
            is_metric=True
        )

I commented out # target_module=CluesToQBQuestion because I'm getting this issue

[YichiRockyZhang]

@tom-doerr Could you please elaborate on how you fixed this since it looks like you've fixed this problem (issue)?

I don't quite understand what is meant by "updated the signature class (target_module)".

[tom-doerr]

Suggest takes the attribute target_module:

    dspy.Suggest(is_assessment_yes(contact_person_assessment.assessment_answer), "The text contains names and titles not in the context. Please revise for accuracy.", target_module=GenerateMail)

I believe I renamed GenerateMail to something else. If I remember correctly I just updated the class name.

[YichiRockyZhang]

Interesting thank you @tom-doerr ! I'll take a look and update this issue if I find anything.