Migration (epic)

[curran]

• [x] Sketch incremental algorithm
• [x] Verify V2 connection
• [x] Refactor to use new refactored DB connection infra
• [x] Verify new environment variable got deployed with new task definition revision
• [x] Verify V3 connection
• [x] POC for Computing OpenAI embeddings
• [x] Redis cloud setup
• [x] Verify Redis connection
• [x] Incremental iteration framing
• [X] Store embedding in Redis
• [x] Compute OpenAI embeddings
• [x] Migrate primordial viz properly
• [X] Migrate a viz
• [x] Migrate associated Users
• [x] Migrate associated Upvotes
• [x] Migrate associated Collaborators
• [x] KNN to backfill forkedFrom
• [x] Observe the result by loading a viz page in beta.vizhub.com
• [x] Migrate a second viz
• [x] Migrate 3 vizzes
• [x] Migrate 4 vizzes
• [x] Run first migration batch by week
• [x] Migrate 100 vizzes
• [x] #198
• [x] #148
• [x] Store embeddings in MongoDB
• [x] #201
• [ ] Migrate revision history from ops (15-min batches)
• [x] Prod mongodb index setup see https://www.mongodb.com/developer/products/atlas/semantic-search-mongodb-atlas-vector-search/ https://www.mongodb.com/docs/atlas/atlas-search/field-types/knn-vector/
• [ ] Migrate associated Permissions from V2 Collaborators
• [ ] Migrate associated Analytics events
• [ ] Store batch number in MongoDB (not Redis, as that's volatile)
• [ ] Estimate MongoDB Storage for Total Migrated DB STORAGE SIZE: 5MB for 1191 vizzes
• [ ] Start cron job to migrate one week's worth of vizzes every day
• [ ] Date histogram of # of ops per month? - sourced from first V3 migration? Easy! Get timestamps of all commits

[curran]

Decide how to compute embeddings

https://observablehq.com/@huggingface/sentence-embeddings-and-dimension-reduction-in-the-browse

OpenAI API?

[curran]

https://github.com/go-skynet/LocalAI

[curran]

Stealth prototype of OpenAI embeddings is done, works great!

[curran]

https://github.com/enjalot/latent-scope-pipeline/blob/main/supabase/functions/csv-embedder/index.ts

[curran]

Current issues:

• Creates a new commit for the primordial viz every time
• Need to handle re-migration of primordial viz in case it's edited

[curran]

Next steps:

Ensure we can run npm run migrate over and over without it causing damage Do not re-migrate already migrated vizzes in the current batch - add tests for this Complete tests for migrating first batch with 3 vizzes Implement forking and commits setup for non-primordial vizzes Use simplified model where all vizzes get only 2 commits - the fork commit, and the most recent change commit Migrate first batch to production Migrate second batch to production