Strange output with starter code

[karthik-rao-delta]

I have been experimenting with dspy for some of my use-cases and notice that I get different behavior than what is presented in the starter code:

model = dspy.OpenAI(model='gpt-4-1106-preview')
dspy.settings.configure(lm=model)

class Emotion(dspy.Signature):
    """Classify emotion among sadness, joy, love, anger, fear, surprise."""
    sentence = dspy.InputField()
    sentiment = dspy.OutputField()

sentence = "i started feeling a little vulnerable when the giant spotlight started blinding me"  # from dair-ai/emotion

classify = dspy.Predict(Emotion)
classify(sentence=sentence)

Prediction(
    sentiment='Sentence: i started feeling a little vulnerable when the giant spotlight started blinding me\nSentiment: Fear'
)

Any advice to get desired behavior?

[karthik-rao-delta]

dspy-ai version 2.1.10

[okhat]

Try dspy.ChainOfThought and/or try optimizing with one of the teleprompters

This behavior is currently indeed common with GPT-4-turbo when used zero-shot via dspy.Predict but it gets better with optimization and/or dspy.ChainOfThought

Closing since it's not really an issue with any parts per se but we'll try to get a nicer zero-shot parser soon

[DSLituiev]

having similar issue with gpt3.5-turbo-16k with Chain of Thought; fields are not properly parsed

[lmiller-phdata]

I think it might be ambiguity with the prompt. Here's an example from the documentation using llama3. You can see the inspect_history shows that it follows the format very explicitly, but the prompt already supplied the Sentence.

In [34]: model = dspy.OllamaLocal(model="llama3:8b-instruct-q6_K", base_url="http://localhost:11434", max_tokens=4000, timeout_s=480)

In [35]: dspy.configure(lm=model)

In [36]: sentence = "it's a charming and often affecting journey."  # example from the SST-2 dataset.
    ...:
    ...: classify = dspy.Predict('sentence -> sentiment')
    ...: classify(sentence=sentence).sentiment
Out[36]: "Sentence: it's a charming and often affecting journey.\nSentiment: Positive"

In [37]: model.inspect_history()

Given the fields `sentence`, produce the fields `sentiment`.

---

Follow the following format.

Sentence: ${sentence}
Sentiment: ${sentiment}

---

Sentence: it's a charming and often affecting journey.
Sentiment: Sentence: it's a charming and often affecting journey.
Sentiment: Positive