search

danielmiessler / fabric

fabric is an open-source framework for augmenting humans using AI. It provides a modular framework for solving specific problems using a crowdsourced set of AI prompts that can be used anywhere.
https://danielmiessler.com/p/fabric-origin-story
22.13k stars 2.31k forks source link

[Question]: What model was used for the example command 3 "extract_wisdom" from the Quick Start? #479

Closed estevao closed 2 weeks ago

estevao commented 3 months ago

What is your question?

I've installed fabric recently and I'm really enjoying, great project!

The only issue I'm having so far is when trying to extract_wisdom from long youtube videos, like the one from example command #3: yt --transcript https://youtube.com/watch?v=uXs-zPc63kM | fabric --stream --pattern extract_wisdom

It's failing with all the models available from OpenAI, example:

$ yt --transcript https://youtube.com/watch?v=uXs-zPc63kM | fabric --stream --pattern extract_wisdom Error: Error code: 429 - {'error': {'message': 'Request too large for gpt-4o in organization org-B3nnupTyTi7ianEA0rN8QRP3 on tokens per min (TPM): Limit 30000, Requested 31111. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}} Error code: 429 - {'error': {'message': 'Request too large for gpt-4o in organization org-B3nnupTyTi7ianEA0rN8QRP3 on tokens per min (TPM): Limit 30000, Requested 31111. The input or output tokens must be reduced in order to run successfully. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}

I was able to summarise this video with llama3 locally but the summary is not as good as gpt4 and the likes. Did something change recently in the OpenAI side or were you using a different model to make it work? Thank you

blade1981m commented 3 months ago

I have been running into that response quite a lot lately (albeit not from OpenAI) so figured I would comment in case it helps. The Limit there for your OpenAI account is 30000 tokens in the request. Between the pattern (extract_wisdom in this case) and the transcript from that YouTube video the total is 31111 tokens (roughly 23000 words) but that is above your current limit. As a comparison I was trying to get a transcript from a 30 minute YouTube video and the specific LLM I was using (groq) limited me to only 5000 tokens in a request. It told me that my request was 9056 tokens. I instead tried a 6 minute YouTube video and groq was able to handle that.

As far as how to resolve, using a paid (or free) publicly provided API often means paying for a higher tier that has higher token limits or switching provider. An alternative would be to run a local LLM (like you tried with llama3) but then you often don't get anywhere near as good a results.

estevao commented 3 months ago

Thanks blade1981m. I wasn't aware of the different Paid tiers, it's well explained here https://platform.openai.com/docs/guides/rate-limits

Paying $50 moved my account to Tier 2 and it's all working now for these long youtube videos. Hope this helps other folks as well.