bpblanken
commented
4 months ago A few reasons for this!
- Project and family tables have the exact same data in them. The family tables are just a secondary index, split out as separate tables because hail doesn't support secondary indexes.
- This didn't happen sooner because our original structure of the project table did not have a good notion of families, it was just a list of all samples in the project. We also set up the DAG so that family and project tables could be created at the same time, with the expectation that the concurrency would make things run faster. However the family tables take so long to write (because of lots of copying, reading the pedigree, hail overhead), that we're better off just reading from the project table (smaller than the subsetted callset) and waiting till the project table is created before making the family tables.
Our update to the project table structure made this incredibly straightforward (and it should have happened months ago 🤦 !)