search

mkuzma96 / DTA

4 stars 2 forks source link

Reproducing Figure 3 #1

Open prockenschaub opened 2 years ago

prockenschaub commented 2 years ago

I have read your paper with great interest and was looking into running the code in this repository on my own machine. Unfortunately, when running the code in sim_exp.py (or main.py) out-of-the-box I am unable to do so. Instead of a replicate of Figure 3a (marginal structural model), I get the following results:

fig3a

Two main differences immediately jump out:

  1. There is little difference between the three models i) with proxy confounders X only (base), ii) DTA deconfounded proxies (dta), and iii) an oracle with access to the true confounders L (oracle).
  2. The absolute value of MSE is similar to that reported in the paper for no confounding (gamma=0) but is much lower than reported for gamma=0.4 or gamma=0.8

Due to computation time, I've only done full 10 seeds 50 hyperparameter search for MSM with the above three levels of gamma and rank=5 but I get comparable results on individual tries with different gammas, ranks and with RMSN.

Do you maybe have any idea as to what might be going wrong?

mkuzma96 commented 2 years ago

Hi, thanks for the interest. I can't say what could've gone wrong off top of my head, I will take a look.