Open Viqsi opened 1 month ago
And yes, hypothetically #229 might also help a little (by presumably taking any equivalence group AECs out of cohortInds), but 1) that's even further out into the Aspirational weeds 2) it's subject to further potential twists given the aspiration for longitudinal data support 3) it doesn't actually solve the problem, just kicks it down the road a bit.
So no citing this as justification for that. No matter how tempting and shiny it may sometimes appear. That thar's fool's gold.
Right now, in our testing environments, creating new cohorts is still relatively fast, but deleting existing cohorts is getting a little oppressively slow, mostly because of the increasing size of the cohortInds table. This might be a thing that's addressed more effectively by Throwing Hardware At The Problem; alternatively, since we already have the need for some kind of GC, we could possibly include cohortInds in that by not deleting from that table at cohort deletion time, and instead periodically removing any rows in that table with cohortIds that can't be found in the
cohorts
table.Something to think about if the problem persists.