search

rmusser01 / tldw

tl/dw (Too Long, Didn't Watch): Your Personal Research Multi-Tool - a naive attempt at 'A Young Lady's Illustrated Primer'
https://tldwproject.com
Apache License 2.0
395 stars 12 forks source link

Removed Chats from Media DB (BREAKING CHANGE - moved to RAG QA db), Added mindmap viewing, token counts in convos, anki flashcard validation #409

Closed rmusser01 closed 3 weeks ago

rmusser01 commented 4 weeks ago

Split chats from the Media DB so that chats about media items are now stored in the RAG_QA db. This allows the media DB to solely contain media items/info about them while the RAG_QA db will store all non-rp chats. This furthers the idea of isolation between all components for re-usability and plug-n-play.

Added Mindmap viewing in Gradio from PlantUML specification. Generate a spec, copy/paste it in, and you got a mindmap.

Added token counts to conversations, rough approximation (not actually tokenizing) to get an idea of how long your convo is.

Added an Anki flashcard validation(working)/viewing(broken) functionality so that you can validate(?) your deck is good(?) Plan is to look at implementing generation with genanki and LLMs so that you can be guided through/ask the LLM to help generate a structured output to be able to create anki flashcards that are exportable/reviewable.

rmusser01 commented 4 weeks ago

Won't merge until I have a migration script to make it easy for people who have chats they want to save.

I recognize its a SQLite DB and one could just perform raw queries to extract the data, I want to make it as easy as possible since not all users are that technical.

rmusser01 commented 3 weeks ago

Need to validate the export/migration script works, besides that this should all be working. Biggest thing is that you can't make a clean upgrade to this version due to DB Schema changes and deletions. (And that my first attempt at an automatic migration failed). So with that being said, I've gone ahead and created a migration script that one can run that will migrate the entries in the prior media DB into the new media DB, and export any/all conversations stored in the old Media DB as markdown files.

Need to test it/confirm it works, but once I do, new features!