tl/dw

Public Demo on HuggingFace Spaces - Demo is broken due to a bug in Huggingface spaces/Gradio

• Please Note: YouTube blocks requests from the demo. You have to provide a logged-in session cookie to bypass it :frowning_face:
• Placeholder content is included for the demo. HuggingFace API is also setup in it, so you can select that as your API.)
• HEADS UP: If you're updating from a prior version, your Media DB is not compatible with the new version. You'll need to start fresh.
  • I've written a script to help you migrate your data from the old DB to the new one. Helper_Scripts/DB-Related/migrate_db.py.
  • Process to migrate your data:
    1. Install/run the new version of the app. This will create a new media_summary.db file in the Databases directory.
    2. Run the migrate_db.py script with the old and new DB paths as arguments. - python migrate_db.py --source media_summary_old.db --target media_summary_new.db --export-path .\
    3. This will migrate all your data from the old DB to the new one, and export the saved conversations to the export-path you specify.
    4. Re-import any/all saved conversations into the new RAG_QA_Chat.db

Screenshot of the Frontpage

Key Features:

• Ingest(Transcribe/convert to markdown) content from (multiple) URLs or local files (video, audio, documents, web articles, books, mediawiki dumps) -> Summarize/Analyze -> Chat with/about the content.- Build up a personal knowledge archive, then turn around and use the LLM to help you learn it at a pace your comfortable with.
• Full Plaintext & RAG Search Capability Search across all ingested content via RAG or 'old-fashioned non-LLM search' (RAG being BM25 + Vector Search/Contextual embeddings + Re-ranking + Contextual Retrieval).
  • Search by content, title, author, URL, or tags, with support for meta-tags, so that you can have the equivalent of 'folders' for your content (and tags).
  • If you'd like to see my notes on RAG: see ./Docs/RAG_Notes.md
  • Notes support, ala NotebookLM, so you can keep track of your thoughts and ideas while chatting/learning, with the ability to search across them or use them for RAG.
• Local LLM inference for offline usage and chat - via llamafile/HuggingFace Transformers.
• 4 Different Chat UI styles - Regular chat, Stacked chat, Multi-Response chat(1 Prompt, 3 APIs) and 4 Separate API chats on one page.
• Local Embeddings Generation for RAG Search - via llamafile/llama.cpp/HuggingFace Transformers.
• Also writing tools! Grammar/Style checker, Tone Analyzer, Writing editor(feedback), and more.
• Full Character Chat Support - Create/Edit & Import/Export Character Cards, and chat with them.
• Arxiv API Integration - Search and ingest papers from Arxiv.
• Chat Workflows - A way to string together multiple questions and responses into a single chat. - Use it to create a 'workflow' for a specific task. Configured via a JSON file.
• Import Obsidian Notes/Vault - Import Obsidian Vaults into the DB. (Imported notes are automatically parsed for tags and titles)
• Backup Management - A way to back up the DBs, view backups, and restore from a backup. (4 SQLite DBs: Media, Character Chats, RAG Chats, Embeddings)
• Trashcan Support - A way to 'soft' delete content, and restore it if needed. (Helps with accidental deletions) - Trashcan is only for the MediaDB.
• Support for 7 Local LLM APIs: Llama.cpp, Kobold.cpp, Oobabooga, TabbyAPI, vLLM, Ollama, Aphrodite, Custom OpenAI API.
• Support for 8 Commercial APIs: Claude Sonnet 3.5, Cohere Command R+, DeepSeek, Groq, HuggingFace, Mistral, OpenAI, OpenRouter.
• Local Audio Recording with Transcription - Record audio locally and transcribe it.
• Structured Prompt Creation and Management - Create prompts using a structured approach, and then edit and use them in your chats. Or delete them.
  • Also have the ability to import prompts individually or in bulk. As well as export them as markdown documents.
  • See ./Docs/Prompts/ for examples of prompts. and ./Docs/Propmts/TEMPLATE.md for the prompt template used in tldw.
• Features to come: Anki Flashcard Deck Editing (Creation is in), Mindmap creation from content(currently in under Utilities, uses PlantUML), better document handling, migration to a FastAPI backend(Gradio is a placeholder UI), and more.

  The original scripts by the-crypt-keeper for transcribing and summarizing youtube videos are available here: scripts here

Table of Contents

• What? | Quickstart | Setup | Using tldw | What's in the Repo / Pieces | Helpful Terms and Things to Know | Setting up a Local LLM Inference Engine | Credits | Roadmap Plans |

Quickstart

Overview of what tl/dw currenlty is

Setting it up Manually

More Detailed explanation of this project (tl/dw)

Planned Features

You can view the full roadmap on the GitHub Issues page.

Local Models I recommend

Command Line usage:

Using tldw

Helpful Terms and Things to Know

Potential Issues

Setting up a Local LLM Inference Engine

• See 'Setting up a Local LLM' in the Docs folder for more information.

Pieces & What's in the original repo?

• See 'Pieces' in the Docs folder for more information.

Similar/Other projects:

• https://github.com/Dicklesworthstone/bulk_transcribe_youtube_videos_from_playlist/tree/main
• https://github.com/akashe/YoutubeSummarizer
• https://github.com/fmeyer/tldw
• https://github.com/pashpashpash/vault-ai <-- Closest I've found open source to what I'm looking to build
• https://github.com/bugbakery/transcribee
• https://github.com/fedirz/faster-whisper-server
• https://github.com/transcriptionstream/transcriptionstream
• https://github.com/lifan0127/ai-research-assistant
• Open Source:
  • https://github.com/lfnovo/open_notebook
• Commercial offerings:
  • Bit.ai
  • typeset.io/
  • https://www.getcoralai.com/
  • https://getcahier.com/#features

  • https://msty.app/

Credits

• The original version of this project by @the-crypt-keeper
• yt-dlp
• ffmpeg
• faster_whisper
• pyannote
• Thank you cognitivetech for the summarization system prompt: https://github.com/cognitivetech/llm-long-text-summarization/tree/main?tab=readme-ov-file#one-shot-prompting
• Fabric
• Llamafile - For the local LLM inference engine
• Mikupad - Because I'm not going to write a whole new frontend for non-chat writing.
• The people who have helped me get to this point, and especially for those not around to see it(DT & CC).

And because Who doesn't love a good quote or two? (Particularly relevant to this material/LLMs)

• I like the lies-to-children motif, because it underlies the way we run our society and resonates nicely with Discworld. Like the reason for Unseen being a storehouse of knowledge - you arrive knowing everything and leave realising that you know practically nothing, therefore all the knowledge you had must be stored in the university. But it's like that in "real Science", too. You arrive with your sparkling A-levels all agleam, and the first job of the tutors is to reveal that what you thought was true is only true for a given value of "truth". Most of us need just "enough" knowledge of the sciences, and it's delivered to us in metaphors and analogies that bite us in the bum if we think they're the same as the truth.
  • Terry Pratchett
• The first principle is that you must not fool yourself - and you are the easiest person to fool. *Richard Feynman