search

wmay / dwnominate

An R interface to the DW-NOMINATE roll call scaling program
https://wmay.github.io/dwnominate
Other
22 stars 4 forks source link

Can't reproduce official DW-NOMINATE scores #7

Closed wmay closed 3 years ago

wmay commented 5 years ago

Given the same starting estimates and votes, dwnominate produces the same output as the DW-NOMINATE fortran code provided by Keith Poole here. However, the official DW-NOMINATE scores still cannot be fully reproduced because we don't know how the starting estimates are calculated. This means dwnominate can't extend the official DW-NOMINATE scores to cover recent sessions of congress, which would be extremely useful.

If the people responsible for the official DW-NOMINATE scores are reluctant to explain the starting estimates, maybe the process can be reverse-engineered.

wmay commented 5 years ago

I just added a common_space function based on the description of the common space procedure in Spatial Models of Parliamentary Voting. This is the procedure used to calculate starting estimates for the official scores (at least according to the book), so this gets us a lot closer to being able to reproduce official scores, at least approximately.

The main remaining obstacle is the spatially unstable periods in US congressional history. Keith Poole says they calculated common space scores for the stable periods and then pieced together the results, but doesn't provide any more detail that I've seen.

I don't expect to exactly reproduce the official scores, but I think if we find a reasonable way to "piece together" the common space scores from the different periods, that would be good enough for researchers to use the resulting DW-NOMINATE scores as a substitute for the official scores.

HowardLR commented 5 years ago

One reason you may be unable to exactly reproduce current DW=NOMINATE scores on voteview.com is that the UCLA folks do not reestimate everything from 1789 but patch on to estimates at a certain date.

wmay commented 5 years ago

I actually stumbled across an explanation of that recently! In the paper "Measuring Bias and Uncertainty in DW-NOMINATE Ideal Point Estimates via the Parametric Bootstrap". So that's the approach I'm following now.

It requires a few changes to my code:

Right now the roll call starting values aren't used, and from what I understand they usually don't make much of a difference to the final estimates. (But correct me if I'm wrong.) For replication purposes I'd like to have that working, though.

And once I get that organized I'll try reproducing the scores again.

HowardLR commented 5 years ago

Did you see my earlier direct e-mail, outside of github (I just joined) on the dimensions of our data. I have a suspicion there is an upper bound from a dimension statement on the number of roll calls per Congress. Thanks, Howard Howard Rosenthal Politics Dept., NYU 19 W, 4th Street, 2nd Floor New York, NY 10012

On Sat, Oct 5, 2019 at 5:58 PM Will May notifications@github.com wrote:

I actually stumbled across an explanation of that recently! In the paper "Measuring Bias and Uncertainty in DW-NOMINATE Ideal Point Estimates via the Parametric Bootstrap" https://doi.org/10.1093/pan/mpp005. So that's the approach I'm following now.

It requires a few changes to my code:

  • I just updated the dwnominate function to allow DW-NOMINATE estimates as starting values.
  • I also want to set up the roll call starting estimates so that they get passed to DW-NOMINATE.

Right now the roll call starting values aren't used, and from what I understand they usually don't make much of a difference to the final estimates. (But correct me if I'm wrong.) For replication purposes I'd like to have that working, though.

And once I get that organized I'll try reproducing the scores again.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/wmay/dwnominate/issues/7?email_source=notifications&email_token=ANM2FBNUIGTANGW3PDJQPH3QNEEWZA5CNFSM4HJOLDZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAN4VUI#issuecomment-538692305, or mute the thread https://github.com/notifications/unsubscribe-auth/ANM2FBMZTQSTUBN4T2YTUGLQNEEWZANCNFSM4HJOLDZQ .

wmay commented 3 years ago

I definitely can't exactly reproduce the scores, if only due to minor code and compiler differences. And I'm not planning to implement the patching-on method used by Voteview any time soon. So I guess I'll have to find some other way to approach or frame this issue.