the result changes across different runs, how to interpret the result?

[hzhz2020]

Dear Authors Thanks for your great work. I am trying to use the DeepIVEstimator to estimate treatment effect for a dataset. I am getting different estimated treatment effect each time I run the estimator. Do you have any suggestion on how to interpret the outputs (i.e., the estimated treatment effect). Shall we average the result across multiple runs?

from econml.deepiv import DeepIVEstimator treatment_model = ... response_model = ... keras_fit_options = ... deepIvEst = DeepIVEstimator(...)

deepIvEst.fit(Y=our_Y,T=our_T,X=our_X,Z=our_Z) treatment_effects = deepIvEst.effect(X = our_X_test, T0 = T0, T1 = T1)

Thank you!

[kbattocchi]

Sorry for the late response. Unfortunately it's difficult to know how to advise you without more information. Here are a few thoughts:

• In general, there is some randomness involved, so you should expect results to vary somewhat. You can see this, for example, if you repeatedly fit the model in our Deep IV notebook against the same data and plot the results. There will generally be a pretty good fit, but there's some variation. If this is the case then averaging over multiple runs seems reasonable.
• However, if your results are very different from run to run, then this probably indicates that something's wrong. Some possibilities are
  1. You don't have enough data, so there is no hope of fitting a good model.
  2. You haven't chosen the best architecture for your neural networks.
  3. Your Z does not meet the requirements of an instrument for this estimator. Deep IV requires even stronger conditions on the instrument than traditional IV estimators do (but the conditions are somewhat hard to state in simple terms). If this isn't the case, then even in the limit as the number of samples goes to infinity, and even if you pick an ideal neural network architecture, then there will still be more than once possible effect function that's consistent with the data and so you could see very different results from run to run.

Without knowing more specific details about your problem, it's hard to know which case might apply to you.

[hzhz2020]

Thank you for your very insightful comments!