search

xuyiqing / fect

fixed effect counterfactual estimators
Other
53 stars 20 forks source link

Very different results estimating the same model using FECT and GSYNTH #16

Closed ccepeda10 closed 1 year ago

ccepeda10 commented 3 years ago

Hello, I noticed that I got radically different results using gsynth and fect for the same model. It's a two way model with the MC method. As the specifications of the models are identical, I was wondering what could be the cause of said difference.

Results using gsynth imagen

Results using fect imagen

I hope you can help me. Cheers.

xuyiqing commented 3 years ago

Thank you. What if you specify the same tuning parameter lambda? It may have something to do with the different default setup for cross-validation.

On Wed, May 5, 2021 at 5:17 PM ccepeda10 @.***> wrote:

Hello, I noticed that I got radically different results using gsynth and fect for the same model. It's a two way model with the MC method. As the specifications of the models are identical, I was wondering what could be the cause of said difference.

Results using gsynth [image: imagen] https://user-images.githubusercontent.com/47113950/117224719-452b3c00-add6-11eb-8200-85a7c443eaf5.png

Results using fect [image: imagen] https://user-images.githubusercontent.com/47113950/117224768-60964700-add6-11eb-8a55-90a7b99008bf.png

I hope you can help me. Cheers.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/xuyiqing/fect/issues/16, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2PKGF6WRIYJT3LUP55AHDTMHNYZANCNFSM44F7S2WA .

ccepeda10 commented 3 years ago

It's 0.1836201 in fect and 1487.01 in gsynth. That's very different. Can a high lambda be a symptom of overfit?

xuyiqing commented 3 years ago

Could you send me (my email) a sample dataset and code for us to replicate the results?

On Wed, May 5, 2021 at 5:42 PM ccepeda10 @.***> wrote:

It's 0.1836201 in fect and 1487.01 in gsynth. That's very different. Can a high lambda be a symptom of overfit?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/xuyiqing/fect/issues/16#issuecomment-833143716, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2PKGB3SVLJKPHI4LA2LCLTMHQYTANCNFSM44F7S2WA .

-- Yiqing Xu

Assistant Professor Department of Political Science University of California, San Diego http://yiqingxu.org/

xuyiqing commented 3 years ago

I will do some tests with simulated data.

On May 5, 2021, at 8:32 PM, Yiqing Xu @.**@.>> wrote:

Thank you. What if you specify the same tuning parameter lambda? It may have something to do with the different default setup for cross-validation.

On Wed, May 5, 2021 at 5:17 PM ccepeda10 @.**@.>> wrote:

Hello, I noticed that I got radically different results using gsynth and fect for the same model. It's a two way model with the MC method. As the specifications of the models are identical, I was wondering what could be the cause of said difference.

Results using gsynth [imagen]https://user-images.githubusercontent.com/47113950/117224719-452b3c00-add6-11eb-8200-85a7c443eaf5.png

Results using fect [imagen]https://user-images.githubusercontent.com/47113950/117224768-60964700-add6-11eb-8a55-90a7b99008bf.png

I hope you can help me. Cheers.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/xuyiqing/fect/issues/16, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB2PKGF6WRIYJT3LUP55AHDTMHNYZANCNFSM44F7S2WA.

ccepeda10 commented 3 years ago

I sent them to your e-mail. Thanks.