LabelFewShot and Dspy.Suggest

[chiragshah285]

Can I use LabeledFewShot with dspy suggestion? I have a bunch of labeled input/output pairs already. I want to compile use that program and do assertions on newly generated outputs. Am I thinking about this the right way? Since I have labeled example already I just need ensure the new outputs from the model are generated per a rule set.

Here is what I'm looking to do:

kbb_page_content = '''
Several things can caus ..
'''
carparts = '''
Codes that illuminate the MIL (Check Engine)
'''
rp_page_content = '''
OBD-II Code P0128 is
''''
KEYWORD ="p0128 code"
fewshot_examples = [
    dspy.Example(keyword=KEYWORD, page_content=kbb_page_content, title="p0128 Code - Coolant Thermostat (Coolant Temperature Below Thermostat"),
    dspy.Example(keyword=KEYWORD, page_content=carparts,title='P0128 Code: Coolant Temperature Below Thermostat Regulating Temperature')
]

class GenerateTitle(dspy.Signature):
    """Generate an SEO-friendly title based on a keyword and page content."""
    keyword = dspy.InputField(desc="The main keyword to include in the title")
    page_content = dspy.InputField(desc="The content of the page")
    title = dspy.OutputField(desc="The generated SEO-friendly title")

# Define the DSPy module
class SEOTitleGenerator(dspy.Module):
    def __init__(self):
        super().__init__()
        self.generate_title = dspy.ChainOfThought(GenerateTitle)

    def forward(self, keyword, page_content):
        title = self.generate_title(keyword=keyword, page_content=page_content).title

        # # Assertions
        dspy.Suggest(keyword_density(title, keyword) >= 2, "Keyword density should be at least 2%")
        dspy.Suggest(title_length(title) <= 60, "Title length should be less than or equal to 60 characters")
        dspy.Suggest(title_readability(title) >= 60, "Title readability should be at least 60")
        dspy.Suggest(title_emotion(title)[0] >= 0, "Title should have a positive sentiment")
        dspy.Suggest(title_relevance(title, page_content) >= 0.5, "Title relevance should be at least 0.5")

        return dspy.Prediction(title=title)

seo_title_suggest = SEOTitleGenerator().activate_assertions()

trained = LabeledFewShot().compile(student=seo_title_suggest, trainset=fewshot_examples)
trained(keyword=KEYWORD,page_content=rp_page_content)

[tom-doerr]

Hi @chiragshah285, to me it seems that it should work (I'm not a maintainer though). Is it not working? Don't know the size of your dataset or what LM you are using, but if you really want to get the best performance, maybe it could make sense to increase the number of samples that LabeledFewShot uses. The default value is 16.

[chiragshah285]

thanks @tom-doerr yeah I was just take 2 examples as a quick test. The problem is more LabelFewShot doesn't support dspy.Suggestion after compilation, from what I see in the code

[arnavsinghvi11]

Hi @chiragshah285 , your code looks correct in terms of using Suggestion within the program. What error do you get while running the code?

DSPy.Suggestions/Assertions are currently compatible with dspy.BootstrapFewShot and dspy.BootstrapFewShotWithRandomSearch, so they should work with dspy.LabeledFewShot as well (which is a candidate within RandomSearch).

[chiragshah285]

thanks @arnavsinghvi11 here is my error message:

Traceback (most recent call last):
  File "/Users/chiragshah/PycharmProjects/dspy_test/seo_title_dspy_metrics.py", line 323, in <module>
    trained(keyword=KEYWORD,page_content=rp_page_content)
  File "/Users/chiragshah/.pyenv/versions/3.11.5/lib/python3.11/site-packages/dspy/primitives/program.py", line 29, in __call__
    return self.forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chiragshah/.pyenv/versions/3.11.5/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 317, in forward
    return wrapped_forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/chiragshah/.pyenv/versions/3.11.5/lib/python3.11/site-packages/dspy/primitives/assertions.py", line 240, in wrapper
    dsp.settings.trace.clear()
    ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'clear'

[chiggly007]

ignore this, this works now! thanks for taking a look