blazespinnaker
commented
3 years ago All the data is not in yet, however looking at NC/FL it's obvious Marist did very poorly and their 'house effect' is so obviously not correct. As did 538, generally speaking. I would suggest doing away with the pollster grading until you get your house in order. Some suggestions:
- Consider splitting it up into a score card, instead of a grade mechanism. This will clarify. The grading mechanism as it stands is condescending and clearly not valid.
- You may already do this, but it would be good to do more. Weight polls more by areas of accuracy. Dig into cross tabs especially. Where did the tabs line up with reality? Where did they not? I know we don't have great exit polling
- Regression! I don't see you talking enough about this. How are you regressing against previous elections? Why aren't you regressing? If you did, 2020 would have been much more accurate.
e-traudt
abitrolly
ScottSupak
Just wanted to say, huge fan of this dataset, huge fan of Nate (bought and loved the book) for like forever. Really hope decency and rationalism wins in 2020.
But, I have some doubts about the pollster rankings, after looking at some data[1][2]. Given that you're being so transparent about this data and I haven't found anything on the web, I assume I'm missing something obvious. Please let me know what it is. Using real math, if you can. I love math. You can't argue with Math!
In particular, I'm looking at Marist College (ranked A+, with 0.2% R house effect!) versus Trafalgar (Ranked C- with R+3.1% ).
I picked these two pollsters as Trafalgar was pretty dramatically right in 2016 and looking at their 2018 results, they caught the crazy squeakers in Florida. Marist is interesting because a lot of polls and ranked highly by 538.
So Trafalgar and that 48 to 46 margin in PA for Trump in 2016. 48 to 44 in FL. Wow. Scott/DeSantis to win in Florida in 2018. It's like they are from the Future.
That said, in 2018, they missed and seemed to over estimate R in some other key states (NV, AZ, went to Ds), but then missed over estimated Dems in OH (went to R). Not sure what the political partisan calculus is there, if there is any, but ok.
Trafalgar also over estimated dems in LA for Gov win in 2018. Not very interesting as it was a 20% margin of victory state (and he did win, after all). But - you have no entry for bias there. Hmmm. I guess if I wanted to fix my house effect I might put my thumb on the scales of polls that don't really matter to me and could help say that I'm fair and balanced? Not sure why you left that one blank.
Now onto Marist. What happened in 2018? Who paid these folks? The average over estimate was like 4-5% in favor of democrats across the board. 50-46 in Florida, dems to win, 4. Dems to win Senate in Indiana! Indiana! They were off by around 8% in favor of Dems, and of course the Rs took it. What was Marist smoking? I want some.
I don't think there is a lot of signal in their call for Ds to underperform in Wicker/Baria Senate race in MS. Nice to see Marist finally show some love to the republicans, but did it have to be in a 20% margin of victory race?
If I were unconsciously biasing polls in a way that helped close races for my political ideology, I probably would unconsciously not care about biasing polls in a way that didn't help in races I didn't care about. So, I like to drop this result in the same that the result for Trafalgar seemed to be dropped (or at least was blank). It seems fair.
Without the Wicker/Baria race, I think the average error is about 5% or so in favor of democrats. With it, around 3% in favor of democrats? Anyways, more than Trafalgar's average error for republicans
To add insult to Injury, Marist called 2016 Florida President General 45.5 to 45 - For Hillary. I mean, come on :)
You know, I am ok with rating Trafalgar poorly. I am sure there is a good reason for that which I won't argue. But relatively speaking, Marist is just as bad if not worse, frankly. Sure, they've been around forever, but results and accuracy matter.