danielmiessler / fabric

fabric is an open-source framework for augmenting humans using AI. It provides a modular framework for solving specific problems using a crowdsourced set of AI prompts that can be used anywhere.
https://danielmiessler.com/p/fabric-origin-story
MIT License
18.75k stars 1.92k forks source link

[Bug]: rate_value calculation is off #581

Open fooman opened 4 weeks ago

fooman commented 4 weeks ago

What happened?

I tried to use the rate_value pattern with the youtube video from the readme running

yt --transcript https://www.youtube.com/watch\?v\=UbDyjIIGaxQ | fabric -sp rate_value

producing the output

{
    "estimated-content-minutes": "97",
    "value-instances": [
        "Introduction of a new AI tool called Fabric, which is open source and reduces friction in using AI.",
        "Demonstration of using Fabric to extract wisdom from a two-hour YouTube video quickly.",
        "Explanation of Fabric's 'Extract Wisdom' feature, showcasing its unique approach to prompt engineering.",
        "Revelation that Fabric's prompts or patterns are both open source and crowdsourced.",
        "Showcase of integrating Fabric with personal projects, like summarizing workout data from Strava.",
        "Introduction to the concept of 'A World of Text' for managing information efficiently.",
        "Tutorial on setting up Fabric on various operating systems, making it accessible to a wide audience.",
        "Use of local AI models with Fabric for privacy and cost-saving benefits.",
        "Integration with Twin Gate for secure remote access to AI tools.",
        "Advanced use cases of Fabric, including creating custom patterns and stitching patterns together."
    ],
    "vpm": "0.10",
    "vpm-explanation": "Calculated by dividing 10 value instances by 97 minutes."
}

The estimated-content-minutes looked too high. I checked the transcript output and that produced 6758 words. According to the instructions from the pattern:

Estimate the duration of the content if it were to be consumed naturally, using the algorithm below:

Count the total number of words in the provided transcript. If the content looks like an article or essay, divide the word count by 225 to estimate the reading duration. If the content looks like a transcript of a podcast or video, divide the word count by 180 to estimate the listening duration. Round the calculated duration to the nearest minute. Store that value as estimated-content-minutes.

this should come to 30 or 38 depending on how it classified the content. This is with GPT-4-0125-preview. Not quite sure if the counting or the math is the issue.

Version check

Relevant log output

No response

Relevant screenshots (optional)

No response

fooman commented 4 weeks ago

Using gpt-4o it comes down to 60 and it also finds the content more valuable.

❯ yt --transcript https://www.youtube.com/watch\?v\=UbDyjIIGaxQ | fabric -m gpt-4o -sp rate_value

{
    "estimated-content-minutes": 60,
    "value-instances": [
        "Introduction of Fabric, an open-source AI tool designed to reduce friction in using AI for problem-solving.",
        "Explanation of how Fabric can extract wisdom from a two-hour YouTube video transcript in moments.",
        "Description of the 'Extract Wisdom' pattern and its open-source, crowdsourced nature.",
        "Demonstration of using Fabric to interact with APIs like Strava for summarizing workout data.",
        "Concept of 'World of Text' for capturing and manipulating all information in text format.",
        "Step-by-step guide on setting up Fabric on different operating systems (Mac, Windows, Linux).",
        "Explanation of using local AI models with Fabric and connecting to remote AI servers.",
        "Introduction to Twin Gate for remote access to AI servers.",
        "Demonstration of stitching patterns together in Fabric for complex tasks like summarizing and writing essays.",
        "Guide on creating custom patterns in Fabric and keeping them private.",
        "Discussion on the philosophy behind Fabric and its goal of augmenting human capabilities.",
        "Insight into Daniel Meer's background and his vision for human flourishing through AI."
    ],
    "vpm": 0.2,
    "vpm-explanation": "Calculated VPM based on 12 value instances over 60 minutes."
}