Report subset bugs - marriage & income

odissei-lifecourse / life-sequencing-dutch

MIT License

0 stars 0 forks source link

Report subset bugs - marriage & income #67

Closed dakota0064 closed 3 months ago

dakota0064 commented 3 months ago

We're seeing bugs in both of these tasks when running one_model_per_year_variable_prediction. Income is showing as all 0s, and marriage is throwing an error because there isn't any overlap with the yearly embeddings / background.

dakota0064 commented 3 months ago

Things we know about the marriage bug

Marriages are correctly represented as pairs of integers.
These integers somehow have no overlap with the embedding keys and the background keys
The embedding subset and background DO have the expected amount of overlap, implying that the issue is not from the subsetting.

dakota0064 commented 3 months ago

The income bug has been resolved, stemming from an issue in the masking during the embedding subset creation.

dakota0064 commented 3 months ago

The marriage bug has also been resolved. We were sampling by individuals, not by pairs, so while we had an embedding overlap of 1,500 individuals per year we still had very few cases where both members of a pair had a valid embedding.