search

time-less-ness / trust-assembly

For SomeGuy's Trust Assembly Project
5 stars 6 forks source link

LLM Data Ingestion into Editable Table #6

Open MelvinSninkle opened 1 month ago

MelvinSninkle commented 1 month ago

Title: Ingest LLM Data into Editable Table Description: Develop a data ingestion pipeline to store new headlines and associated metadata in a structured, easily editable table. Dependencies: #4 Tasks:

  1. Create a database table with fields: URL, old headline, new headline, model used, prompt identifier, date, and time of action.
  2. Build an ingestion pipeline to store LLM-generated headlines and metadata.
  3. Implement edit functionality for easy updates to any field.
  4. Include logging for all data ingestion activities. Acceptance Criteria: • Database table is correctly set up with all specified fields. • Data is ingested accurately and is easily editable. • Successful logging of all ingestion activities and changes. Priority: High Labels: Database, Backend, Data Ingestion, MVP
Melvillian commented 1 month ago

LGTM 👍

Which Database should we use? I think Postgres is the simplest choice because it's so widely used, and has generic-enough text querying capabilities to do what we want.

To make it simpler we can use a hosted solution for Postgres. I've heard good things about Neon, but we could also go with Supabase which I think is more popular.

My suggestion is to go with Supabase using Postgres

MelvinSninkle commented 1 month ago

I am good with your suggestion on the tech side. Do you know what we need to get us off the ground monetarily?

Melvillian commented 4 weeks ago

Looks like $25/month should get us to a point where if we need to pay more than $25, that is a good problem to have!

https://supabase.com/pricing

image

MelvinSninkle commented 3 weeks ago

I can get that set up

On Thu, Oct 24, 2024 at 8:30 AM Alex Melville @.***> wrote:

Looks like $25/month should get us to a point where if we need to pay more than $25, that is a good problem to have!

https://supabase.com/pricing

image.png (view on web) https://github.com/user-attachments/assets/e629deb9-2331-47f4-abea-62f596ed9d3a

— Reply to this email directly, view it on GitHub https://github.com/time-less-ness/trust-assembly/issues/6#issuecomment-2435613486, or unsubscribe https://github.com/notifications/unsubscribe-auth/BL5APYXS5LOZEDVUWOWTF43Z5EHAPAVCNFSM6AAAAABQI6QTXKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZVGYYTGNBYGY . You are receiving this because you authored the thread.Message ID: @.***>