BayesianSignatureOptimizer - missing 1 required positional argument: 'example'

[cognitivetech]

as explained in the title, I'm getting this error from the following code: TypeError: TemplateV2.__call__() missing 1 required positional argument: 'example'

import dspy
from dspy.teleprompt.teleprompt import Teleprompter
from dspy.signatures import Signature
from dspy.evaluate.evaluate import Evaluate

lm = dspy.OllamaLocal(model="mq8", max_tokens=4000, timeout_s=120, temperature=0.7)
# This sets the language model for DSPy.
dspy.settings.configure(lm=lm)

import csv

# Load the dataset from the file
with open('notes-dataset.csv', 'r') as file:
    reader = csv.DictReader(file)
    data = list(reader)

# Convert each entry into a DSPy example
dataset = []
for item in data:
    example = dspy.Example(
        inputs={'document': item['document']},  # Use 'document' column as input
        outputs={'notes': item['notes']},  # Use 'notes' column as output
    )
    dataset.append(example)

# Choose size of each set
train_size = 0.5
dev_size = 0.3
test_size = 0.2
# Calculate the number of examples for each set
num_examples = len(dataset)
num_train = int(num_examples * train_size)
num_dev = int(num_examples * dev_size)
num_test = num_examples - num_train - num_dev

# Split your dataset into train, dev, and test sets
trainset = dataset[:num_train]
devset = dataset[num_train:num_train + num_dev]
testset = dataset[num_train + num_dev:]

print(f"Trainset size: {len(trainset)}, Devset size: {len(devset)}, Testset size: {len(testset)}")

class Notes(dspy.Signature):
    """Bulleted Markdown Notes, Headings and Terms in Bold"""

    document = dspy.InputField()
    notes = dspy.OutputField()

document = """POLYVAGAL SAFETY Attachment, Communication, Self-Regulation STEPHEN W. PORGES A NORTON PROFESSIONAL BOOK+ Introduction: An Embedded History of a New Science+ When the Polyvagal Theory was initially conceptualized, I did not consider it as an expansive paradigm-shifting model. In developing it, I thought it was consistent with antecedent research and theory. In a way, I thought it was the obvious next step. During the 25 years following the initial presentation of the Polyvagal Theory, through my experiences with the scientific community across many disciplines, I learned that the theory provides the framework for a new brain-body or mind-body science. Moreover, I learned that as health-related disciplines embraced the theory, a new polyvagal-informed strategy could be embedded in mental and physical health treatment models. This strategy would focus on recruiting the nervous system of the client or patient as a collaborator on a shared journey toward wellness. Embracing this strategy would emphasize that the body’s quest for safety is embedded in the actions of the nervous system in promoting health. Thus, safety and threat are not only psychological constructs, but have parallels in the autonomic nervous system. When the autonomic nervous system is in a state of calmness and social accessibility, then treatments for both mental and physical disorders can be efficiently implemented. In contrast, when the autonomic nervous system is in a state of defense and vulnerability, then it is in an antagonistic state that blunts the effectiveness of treatment. Simply stated, the theory uncovered the structures and portals through which our neurobiological quest for safety through connectedness with others may be implemented. When a new theory is presented, its intellectual accessibility is dependent on several complex and often not acknowledged historical features. Most importantly, there is a need to understand (1) antecedent theories, (2) the language used to describe the hypothetical constructs illustrating the functions described in the theory, and (3) questions that the theory proposes to answer. Since evolution is an organizing principle in Polyvagal Theory, in which neuroanatomical structures are proposed to facilitate specific adaptive functions, an understanding of vertebrate evolution and neuroanatomy of the neural regulation of the autonomic nervous system are required for a critical dialogue. In addition, since the theory leads to testable hypotheses based on dynamic adjustments in autonomic function, a sophisticated understanding of the metrics of measurement (e.g., time series analyses) is necessary. Scientists are intellectually focused and affiliated with peer groups defined by a shared commitment to study common problems. They see the world through their research questions, explicit and implicit theories, and methodologies. My scientific origins are rooted in the world of psychophysiology, a science that emerged in the 1960s to study how physiological responses paralleled psychological phenomena such as mental effort, attention, expectancy, stimulus detection, decision making, truthfulness, and preference. At the time I first presented the Polyvagal Theory, arousal theory was prevalent in psychophysiology. Although arousal theory had a long, influential history in science, it had a relatively simplistic underlying model. Basically, arousal theory emphasized that arousal was a linear construct indexing a dimension from low to high levels of activation that could be measured or inferred from observing behavior or physiology. The relationship between arousal and performance was often portrayed as an inverted U-shaped function in which optimal performance occurred within a midlevel range, while poor performance was observed at low and high levels of arousal. This relationship was known as the Yerkes-Dodson law (Yerkes & Dodson, 1908). Metaphorically, arousal represented the energy of the human nervous system. Arousal was easily understood, since when it was reflected behaviorally it could be quantified as greater activity and when reflected autonomically it could be observed as increases in sweating and heart rate. Early psychophysiological research assumed that peripheral autonomic measures provided sensitive indicators of arousal. This view was based on a rudimentary understanding of the autonomic nervous system in which changes in electrodermal activity (e.g., sweating) and heart rate were assumed to be accurate indicators of sympathetic activity. As the activation arousal theory developed, a continuity between peripheral autonomic responses and central mechanisms was assumed (see Darrow et al., 1942), and sympathetic activity was assumed to parallel activation of the brain. According to this assumption, organs influenced by sympathetic efferent fibers, such as the sweat glands, blood vessels, or the heart, were potential indicators of limbic or cortical activity (Duffy, 1957; Lindsley, 1951; Malmo, 1959). Although the specific pathways relating these various levels were never outlined and are still sketchy, electrodermal (e.g., GSR) and heart rate became the primary focus of research during the early history of the Society for Psychophysiological Research. This was due to their presumed sympathetic innervation and, in part, to their measurement availability. By default, this emphasis created a research environment that neglected several important factors: (a) parasympathetic (e.g., vagal) influences, (b) interactions between sympathetic and parasympathetic processes, (c) peripheral autonomic afferents, (d) central regulatory structures, (e) the adaptive and dynamic nature of the autonomic nervous system, and (f) phylogenetic and ontogenetic differences in structural organization and function. Polyvagal Theory proposed a more complex nonlinear model of autonomic regulation that focused on identifiable and potentially measurable neural pathways that contribute to autonomic regulation through a definable feedback system that could promote homeostatic function. The theory required a conceptualization of autonomic regulation that was challenging to psychophysiologists. At the time, psychophysiology emphasized a top-down representation of central nervous system function in measurable peripheral physiology (e.g., autonomic response patterns). Thus, many psychophysiologists hypothesized that systematic quantification of peripheral autonomic responses would provide reliable information linked to brain function and mental processes. Prior to the Polyvagal Theory, my research was consistent with this top-down perspective. For example, I wrote an article titled “Heart Rate Patterns in Neonates: A Potential Diagnostic Window to the Brain” (Porges, 1983)."""

# Baysian signature optimizer
from dspy.teleprompt import BayesianSignatureOptimizer
teleprompter = BayesianSignatureOptimizer()
bayesian_compiled_classifier = teleprompter.compile(Notes(), devset=devset,
                                                optuna_trials_num=3,
                                                max_bootstrapped_demos=4,
                                                max_labeled_demos=4,
                                                eval_kwargs=kwargs)

bayesian_compiled_classifier.save(path="saves/BayesianSignatureOptimizer.json")

[bryonmccoy]

I'm having the same issue. I'd love it if someone would fix it... I've spent about 40 hours on this one problem. I've tried fixing this every which way I can.

[okhat]

cc @XenonMolecule @klopsahlong

[okhat]

Thanks for reporting this @cognitivetech @bryonmccoy . Can we get a full stack trace of the error?

[okhat]

Seeing a full trace of the error stack will probably make this pretty easy to spot

[KCaverly]

I took a look at this, and managed to recreate it with a relatively small subset.

import dspy

class Notes(dspy.Signature):
    """Bulleted Markdown Notes, Headings and Terms in Bold"""
    document = dspy.InputField()
    notes = dspy.OutputField()

Notes()

I am including the Notes(), cause you are passing the Signature directly into the Teleprompt, in this way.

If you take a look at the way __new__ and __init__ are set up for dspy.Signature, it appears that simply declaring the Notes class itself, instantiates the object.

Ie. Calling only the below

import dspy

class Notes(dspy.Signature):
    """Bulleted Markdown Notes, Headings and Terms in Bold"""
    document = dspy.InputField()
    notes = dspy.OutputField()

Will call both the dspy.Signature.__new__, and then dspy.Signature.__init__, without instantiation in the python file directly.

Later on when Notes() is being called, this is not instantiating the Notes object, but rather calling the Notes.__call__ method. It is at this point that __call__ requires an 'example' argument to pass to `TemplateV2'.

If for example, you get around this by Simply passing Notes as opposed to Notes() into the teleprompt, you bypass the Example error, but the teleprompt tries to deepcopy the signature. A deepcopy method is used for dspy.Module but not dspy.Signature.

Should a dspy.Signature be directly passed into Teleprompters?

[XenonMolecule]

Yep I suspect this is the issue. BayesianSignatureOptimizer takes a dspy.Module to optimize. I acknowledge the naming is a bit confusing and we are likely going to update the optimizers soon with the official optimizer release.

Good sleuthing @KCaverly!

For this example making a dspy.Module wrapper for the Notes() signature should resolve this particular problem. You can use either dspy.ChainOfThought or dspy.Predict.

You can either pass dspy.Predict(Notes)() to the optimizer, or create a custom wrapper module:

class WriteNotes(dspy.Module):
    def __init__(self):
        super().__init__()
        self.generate_notes = dspy.Predict(Notes)

    def forward(self, document):
        return self.generate_notes(document=document)