search

d2cml-ai / csdid

CSDID
https://d2cml-ai.github.io/csdid/index.html
MIT License
20 stars 5 forks source link

Add Basic Test Against R::did #39

Closed s3alfisc closed 1 month ago

s3alfisc commented 1 month ago

Hi all,

As suggested in #38, this PR adds a test against R::did via rpy2 and a CI workflow to run the test.

To do so, pytest and rpy2 are added as dependencies to requirements.txt.

Currently, I only implement one simple test of the for the ATE and associated SEs; tests for other aggregation levels could easily be added following the same schema.

Tests for point estimates pass, but not for the SEs, where I observe minor differences. Maybe it is small sample corrections?

pedrohcgs commented 1 month ago

Probably related to bootstrap. We can try comparing the analytical std error.

Pedro H. C. Sant'Anna https://psantanna.com https://psantanna.com

Warning: This email may contain confidential or privileged information intended only for the use of the individual or entity to whom it is addressed. If you are not the intended recipient, please understand that any disclosure, copying, distribution, or use of the contents of this email is strictly prohibited.

On Sun, Sep 22, 2024 at 13:03 Alexander Fischer @.***> wrote:

Hi all,

As suggested in #38 https://github.com/d2cml-ai/csdid/issues/38, this PR adds a test against R::did via rpy2 and a CI workflow to run the test.

To do so, pytest and rpy2 are added as dependencies to requirements.txt.

Currently, I only implement one simple test of the for the ATE and associated SEs; tests for other aggregation levels could easily be added following the same schema.

Tests for point estimates pass, but not for the SEs, where I observe minor differences. Maybe it is small sample corrections?

You can view, comment on, or merge this pull request online at:

https://github.com/d2cml-ai/csdid/pull/39 Commit Summary

File Changes

(6 files https://github.com/d2cml-ai/csdid/pull/39/files)

Patch Links:

— Reply to this email directly, view it on GitHub https://github.com/d2cml-ai/csdid/pull/39, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABE7346L3ALIPMXZ57YAYU3ZX3Z7JAVCNFSM6AAAAABOUW5LGCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGU2DCMJWG42TCMQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

s3alfisc commented 1 month ago

Yep, increasing the # of bootstrap iterations reduces the difference and the tests pass. For some reason, I was assuming that only the CIs were based on a multiplier bootstrap & the SEs were analytical.

alexanderquispe commented 1 month ago

Hi @s3alfisc , thanks a lot for your contribution! Yes, I was doing the same excercise for the difference between the stds and for bootstraps when increasing the # obs the difference reduces, so great that you found the same :). I just merged the commit.