Closed Asentient closed 1 month ago
Have you considered the context limitations? Maybe the extract_wisdom pattern approaches or exceeds the input context window of your chosen local model, or comes close enough that there's not much left for an effective response. You might try creating a custom version(s) of extract_wisdom like extract_wisdom_mini that is focused and condensed on providing specific insights... one of the frontier models could probably help you construct that effectively.
Keep in mind too that for the purposes of something like extract_wisdom, the output of the local models running on a consumer gpu just aren't going to compare to the models working with 70b parameters like Groq, much less the big closed source models, so you have to manage your expectations based on the model's capabilities.
I have considered it before, yes and here are my findings.
I checked that the local and online model are identical in parameters, context window, etc and yet i kept getting vastly different results between them. Notice how i didnt use chatgpt in this discussion for absolute clarity, we are talking about identical models in size, and supposedly settings too unless there is something i missed.
both online and local versions have a context window of 32768
llama_model_loader: - kv 3: qwen2.context_length u32 = 32768
[!info] This indicates that the context length for the local model "Qwen2-7B-Instruct" is 32,768 tokens.
qwen2-7b
└─➞ ytt https://www.youtube.com/watch?v=zHkcnDaCNUU | fabric -m qwen/qwen-2-7b-instruct -sp extract_wisdom
SUMMARY: In summary, OpenAI has outlined its roadmap towards AGI (Artificial General Intelligence) through five levels: Chatbots (Level One), Reasoners (Level Two), Agents (Level Three), Innovators (Level Four), Organizations (Level Five). OpenAI has been worki
IDEAS:
INSIGHTS:
QUOTES: "OpenAI believes we'll move through each level towards true AGI." "Strawberry aims at autonomous deep research capabilities." "Dolly image model improvements enhance text clarity." "Sora showcases impressive capabilities through advanced demos." "HubSpot offers comprehensive resources integrating ChatGPT." "Google VDS simplifies sales training video creation." "Microsoft Designer integrates Co-Pilot sidebar enhancing image creation." "Codstroll Mamba offers faster code generation capabilities." "Amazon Rufus shopping assistant streamlines customer queries within its app."
HABITS:
FACTS:
REFERENCES:
ONE-SENTENCE TAKEAWAY: The rapid advancement in artificial intelligence technologies continues across various sectors including education, shopping assistance, music identification, dental forensics, image generation, sales training videos, autonomous research capabilities,
RECOMMENDATIONS:
HABITS: 1. Regularly updating skills through continuous learning resources such as online courses or workshops related specifically aimed at advancements within artificial intelligence technologies.
2. Utilizing advanced tools such as Sora, Gemini, VDS (Video Design Studio), etc., efficiently by understanding their unique features before implementation within specific projects or workflows.
3. Keeping track daily updates regarding industry trends through following relevant blogs, podcasts, newsletters dedicated specifically towards advancements within artificial intelligence technologies.
4. Integrating artificial intelligence into daily work processes effectively by identifying areas within specific job roles where automation could potentially enhance productivity without compromising quality standards set forth by respective industri
5. Staying informed about regulatory changes affecting technology usage within organizations by regularly attending seminars hosted by legal experts specializing specifically within areas related directly towards artificial intelligence technologies e
FACTS: 1. OpenAI outlines five stages towards AGI development highlighting progression towards achieving true artificial general intelligence capabilities across multiple sectors including education, healthcare, finance among others demonstrating significant [Breaks of into Gibberish here.]
$ ytt https://www.youtube.com/watch?v=zHkcnDaCNUU | fabric -m qwen2:7b-instruct -sp extract_wisdom
In this video, the speaker discusses various updates in the field of AI. They compare performance scores between different models and introduce a new model called GPT 40 Mini, which appears to perform well across several benchmarks compared to other smaller models from competing platforms. The speaker also mentions Nvidia's Mistol Nemo model, designed for local deployment on devices with limited internet connectivity or stringent data privacy requirements.
The video highlights Google's AI sponsorship for Team USA in the Olympics and an AI news channel called Futur Tools where viewers can stay updated on the latest AI news, tools, and research. The speaker also mentions HubSpot as a sponsor of this particular video.
In summary, the key points are:
These updates showcase advancements and developments in AI technology across various domains, including cloud computing, local deployment options for models, sponsorships at major events like the Olympics, and curated content platforms for staying informed on AI-related topics.
What else could be other than context_window? If anyone has any other parameter for me to check that would be helpful.
+1, same for me
I tested this will less/more input data. For less input, it does adhere to the format but with more data, it starts to ignore the format.
It probably has something to do with the token size 🙃 I am wondering if we can feed large data in chunks for it to process and then stitch the response together.
I tried with llama3:8b-instruct-q8_0, it has 8k token size, yet same results.
I can confirm it's the context window.
I tried calling ollama directly with the same system prompt, passing in a higher num_ctx to 18k and using mistral nemo, it gives me much better results for long vids.
Is it possible to pass in ollama flags via the cli?
In another post someone gave a solution that worked for me:
I tried with
FROM llama3:8b-instruct-q8_0
PARAMETER num_ctx 8192
Didn't work. I tried with 6h long video and 55m long video.
I tried with
FROM llama3:8b-instruct-q8_0 PARAMETER num_ctx 8192
Didn't work. I tried with 6h long video and 55m long video.
For me a 24min video i needed minimum 8192. So i guess 6h long video you need a lot more context
Local LLM Output Short, Unstructured and Unformatted Compared to GPT-4
Description:
Users have reported that when using the models locally, the output is significantly shorter and does not follow the prompt instruction or the formatting provided by GPT-4. This issue is consistent across different setups, and different local models including various hardware configurations and operating systems.
Reproduction Steps:
yt --transcript
command with a YouTube video link.fabric -m llama3:latest -sp extract_wisdom
.Observed Behavior:
Expected Behavior:
Local LLM should produce similar comprehensive outputs as GPT-4, or similar subpar responses.
Additional Context:
Discussion Links:
504 Discussion 504
428 Discussion 428
740 Question 740