josh-ashkinaze / plurals

Plurals: A System for Guiding LLMs Via Simulated Social Ensembles
https://josh-ashkinaze.github.io/plurals/
12 stars 2 forks source link

Moderator more flexibility #31

Closed josh-ashkinaze closed 4 months ago

josh-ashkinaze commented 4 months ago

To discuss at weekly meeting. Running some demos I found myself not wanting to use persona and combination instructions---I am not sure why we can't just allow a parallel initialization scheme to normal Agents with system instructions. Consider that in moderate_responses we run this before calling moderator.process():

self.system_instructions = SmartString(
            self.system_instructions).format(
            task=original_task,
            previous_responses=combined_responses_str,
            persona=self.persona) 

So this makes me think that up until this point system instructions can just have placeholders.

But here is the problem: We set as a default variable for persona_template that persona_template is 'default'. And in validation in the Agent class, I raised an error when a user provides both system_instructions and a persona_template. So if I simply add a system_instructions parameter this will raise a ValueError if we keep this default. Now, we can simply catch the error but that's non optimal.

So I am having trouble thinking of how to reconcile these apparently irreconcilable desired behaviors:

  1. We want some kind of default persona for Moderators
  2. We want to raise an error if a user passes in system_instructions and persona_template
  3. If we leave default persona for Moderators, system_instructions throws valueError due to validation in point 2
  4. If we catch ValueError from point 2, this will be inconsistent since Agents would raise exception but Moderators wouldn't---and feels weird.

Maybe a solution is something like this: CASE 1: A user passes in both system instructions and persona --> Raise ValueError CASE 2: A user passes in persona but not system instructions--->Let's set persona_template as default if not set by user CASE 3: A user passes in system instructions but not persona --> Just use that persona_template and persona are None

josh-ashkinaze commented 4 months ago

I did this as part of this new release. see here https://github.com/josh-ashkinaze/plurals/blob/c40ee95f16b5ae0ef58081689731b93f84dd3b51/plurals/deliberation.py#L33