Job Pipeline for vectorization process

[armintalaie]

Overview

Implement job queue system to handle asynchronous processing of uploaded files (vectorization, embedding generation)

When files are uploaded by the user we can't vectorize and generate embeddings for the content or post right there. What the API endpoints can do is insert a job in a new jobs table. Usually a process goes every few hours or so and executes the jobs.

Database Changes

• [x] Add jobs table:

  - id: UUID
  - status: enum (pending, processing, completed, failed)
  - job_type: enum (file_embedding)
  - document_id: UUID (nullable)
  - post_id: UUID (nullable)
  - metadata: JSONB
  - error: text (nullable)
  - created_at, updated_at, processed_at timestamps

Check the example directory for how the jobs might work but basically go the process will go over the record that are not completed.

• If it is file_embedding -> grab the document and file from db and use the documentProcessor class to save the embedding

Components

• [x] Job Creation utility functions

  • Create job record when content uploaded
  • Return job ID for status checking

• [x] Job Processor Script For now just save it in a jobs.py file that every time we call it, it tries to run the not executed jobs.

  • Process pending jobs
  • Update job status

Error Handling

• [x] Retry logic for failed jobs (have a fail threshold -> if fails job 2 times skip job for script)
• [x] Error logging

Tasks:

• [x] Write migrations (used Supabase GUI)
• [ ] Add job creation to upload endpoints; do this task after #72
• [x] Implement processor script

[rayzhou4]

need to add actual job processing logic eventually in jobs.py (currently just logs saying that job processing happens)