danielmiessler / fabric

fabric is an open-source framework for augmenting humans using AI. It provides a modular framework for solving specific problems using a crowdsourced set of AI prompts that can be used anywhere.
https://danielmiessler.com/p/fabric-origin-story
MIT License
25.79k stars 2.74k forks source link

[Bug]: Running `improve_prompt` on a prompt with template variables fails #1171

Open osheari1 opened 6 days ago

osheari1 commented 6 days ago

What happened?

With a prompt, such as improve_security_policy

# IDENTITY and PURPOSE
You are an expert in cybersecurity standards and policies. You specialize in creating clear, well written policies for companies and corporations (for example for certifications such as SOC2 and HiTrust).

You have been tasked with improving an existing security policy that needs to be improved, and you must output an improved report finding in markdown format.

# GOAL
Given a policy type and set of requirements, provide a well written, concise improved security policy in markdown format.

# STEPS
...
# OUTPUT INSTRUCTIONS
...
# INPUT
INPUT:
## Requirements: 
{{reqs}}
## Policy 

piping into improve_prompt returns

$ cat system.md | fabric --pattern improve_prompt 
missing required variable: reqs

Expected behavior: Improve prompt would work on patters with variables / plugins / other future stuff that may be added to the templating features.

I'm assuming this is because variables are validated after stdin is substituted into the template.

Version check

Relevant log output

No response

Relevant screenshots (optional)

No response

mattjoyce commented 4 days ago

hello @osheari1 yes, you're correct.

the user input and pattern are combined an then the template system tries to resolve all double brace pairs. {{ }} A work around might be just to provide a variable. -v=req:""

potentially we could have --novars or something to disable substitution.

osheari1 commented 2 days ago

Hello!

I'm wondering if in the long run, a more robust solution would be needed... :thinking:

Thinking of two cases:

Pattern

# Example Output
// Some example explaining Jekyll syntax
{{ jekyll-var-no-sub-wanted }}

# Output 
{{ some-fabric-variable }} 

Input

Stuff {{ fabric-var-no-sub-wanted }}

Here, {{ fabric-var-no-sub-wanted}} and {{ jekyll-var-no-sub-wanted }} would both throw errors.

Problem

Questions

For example, say you have a pipeline, cat ... | fabric ... | command A | fabric ... | command B | fabric ...: Depending on the task, you won't necessarily know what output the fabric commands will produce. Meaning, in some cases, they may produce instances of {{...}} in intermediary steps, which will then, without the users knowledge, attempt to be substituted by fabric in later calls.

For example,

$ cat stuff.txt | sed 's/{{var}}/subbed/g' | fabric -p do_stuff

Solutions?

mattjoyce commented 1 day ago

I'll admit, I did not consider folk would want to process jekyll templates when I made the template system.
I do use variables in input, it useful and think this will be increasingly useful when extension are available. Probably the simplest approaches to this are

  1. Do not fail if a variable is missing - perhaps a .env setting to disable or enable.
  2. Include a cli option to not process vars. --skiptemplate, --novars or similar.
osheari1 commented 1 day ago

Yeah I think that will work if it applies specifically to INPUT vars.

There are two cases where this issue could pop up:

  1. when we have jekyll vars (as an example) in a prompt This would still be an issue with '--novars'.
  2. when we have jekyll or fabric vars in input This would be solved by --novars, if --novars only applies to INPUT vars only.

Is there something that could be done about the first case as well?

mattjoyce commented 12 hours ago

Honestly I think it's just easier to not fail on missing vars.

osheari1 commented 3 hours ago

I think that would be helpful. For this case, what if we added some debugging information about variable substitution when '--dry-run' (or some '--debug' flag) is passed? If we skip var validation entirely, people may unexpectedly not pass -v=<name>:<val> when they should.

Perhaps something like (pseudo-code)

$ echo "<user> | fabric --dry-run/debug --pattern <pat> -v=name-1:val-1 -v=name-2:val-2
...
variable,value
name-1,val-2
name-2,val-2
# name 3 was in prompt/input but not passed
name-3, <missing>