implement CRN

[fangzhouli]

• Literature review of the original paper and github code
  • paper: Estimating counterfactual treatment outcomes over time through adversarially balanced representations
  • github: https://github.com/ioanabica/Counterfactual-Recurrent-Network
• Reformulate architecture based on our problem (see midterm report for baseline)
  • treatment?
  • outcome?
• Evaluate, compare, visualize
  • qualitative results to compare CRN and traditional RNN

[fangzhouli]

11/24/21, Discussion

1. Data cleaning/preprocessing for CRN (Chongzhou, ETA: 11/26/21)
   • work in progress
2. Data generation (Fang, ETA: 12/01/21)
   • Implementation (Fang, ETA: 11/30/21) -- Fix the length of time series K -- Randomly choose (or all) AFIS data -- Randomly (or using explainability) to identity next actions -- Generate gold standard using LSTM simulator (original paper, 10000 samples) --- Actions for each time stamp --- Outcome for each time stamp --- Input for each tims stamp
   • Testing (Fang, ETA: 12/01/21)
3. CRN modification (Chongzhou, ETA: 12/01/21)
   • Change feature names
   • Removing “V” global metadata
4. CRN training (Matt, ETA: 12/01/21)
   • hyperparameters tuning
5. Evaluation (Fang/Matt, ETA: 12/03/21)
   • What are we comparing with? -- CRN vs. RNN trained on counterfactual data
   • How do we compare -- accuracy

Note: 3-4 days for report Hard Deadline: 12/08/21

[fangzhouli]

Data generation

Finished:

• Basic functionality
  • Specify length of time series and generate counterfactual data.
  • Randomly sample a training project, randomly choosing a feature and an action.
  • Feed the counterfactual new point to the LSTM simulator to generate outcome
  • Return (counterfactual data, outcome, action)

TODO:

• Finish generate 10000 samples (or more? we need to test on CRN) after Wed's discussion
• original data are min-max-ed. counterfactual data might be out of bound, is it an issue?
• perturbing one action might be too trivial to the output, maybe we can try:
  • increase the depth: many interventions (one feature for each intervention), see the cumulative outcome
  • increase the width: Each intervention multiple features (but how do we interpret? will it be too complicated? Or maybe we can group multiple actions together in a way they make human-interpretable sense?)
• Design a qualitative experiment: Find some projects with prediction score that is not too "one-way", e.g., [0.99, 0.01]. It might be the case where the more confident the model is, harder to find perturbations to "change its mind". We should find several non-one-way data points to see direct result of interventions

[fangzhouli]

12/01/21 Discussion Data genetation

• Randomly choosing from 18 features seem to be too challenging
  • maybe decrease the perturbation set size to 4
• Testing all features to see whether individual feature is not good for changing the outcome