search

ProfessorAmanda / econsimulations

This repository provides the code underlying simulations for teaching statistics and econometrics. The project site, which includes teaching materials as well as the link to the simulations, is located here: https://amandagreggeconomics.com/statistics-simulations-project/
Other
1 stars 0 forks source link

Simulate type 1 error #198

Closed ethansaxenian closed 3 years ago

ethansaxenian commented 3 years ago

Populations can now be generated and f-stats are plotted.

How do we want to represent rejecting/accepting? Similar to hypothesis testing where the points are red/green?

What information should we include in the tooltip for each point? I'm thinking f-stat, pval, reject/accept at least.

Still need to add summary text.

ProfessorAmanda commented 3 years ago

Looks so good! I'll have to think about whether there are any minor changes, but I'm really happy with this already. To answer your questions:

  1. Yes, red/green dots is perfect.
  2. Those pieces of info for the tooltip would be great.
  3. I'll keep posted for the summary text addition.
ethansaxenian commented 3 years ago

The points are now color coded accordingly!

ethansaxenian commented 3 years ago

...and summary text is now added

ProfessorAmanda commented 3 years ago

Check the calculation of those p-values. All the red dots should be bunched up at the right tail. Also, given the same sample sizes and the same number of groups, the same F statistics should yield the same p-values, which they currently do not.

ProfessorAmanda commented 3 years ago

I checked the math for the F-stats above. Actually I think the p-values are correct but the SSTR and SSE are somehow calculated incorrectly. Check parentheses in your formulas.

ethansaxenian commented 3 years ago

Check the calculation of those p-values. All the red dots should be bunched up at the right tail. Also, given the same sample sizes and the same number of groups, the same F statistics should yield the same p-values, which they currently do not.

The p-values are calculated using the jStat function... I hope this isn't the problem.

Actually, the jStat function takes a list of arrays (the samples?) which will be different in each replication.

ProfessorAmanda commented 3 years ago

I found the mistake in your F formula. You were using population sizes instead of sample sizes.

ethansaxenian commented 3 years ago

jStat.anovaftest( array1, array2, ...., arrayn ) Returns the p-value of the f-statistic from the ANOVA test on the arrays.

ethansaxenian commented 3 years ago

I found the mistake in your F formula. You were using population sizes instead of sample sizes.

In which calculation? Should it be all of them?

ProfessorAmanda commented 3 years ago

Everywhere you calculate an SSTR or an SSE. The numerators have the sample sizes, not the population sizes.

ProfessorAmanda commented 3 years ago

SSTR = sum(over populations) sample size * (sample mean - overall mean)^2

SSE = sum(over populations) (sample size - 1) * (sample std dev)^2

ProfessorAmanda commented 3 years ago

(Where sample size = sample size drawn from group k)

ethansaxenian commented 3 years ago

Ok, should be fixed. The problem now is that there are a lot of f-stats = 0, which throws off the scale of the plot.

ProfessorAmanda commented 3 years ago

Good! At least it's working in the sense of all red dots to the right. Don't forget to fix the F stats in the top part of the module as well.

As far as the scaling goes, there shouldn't be any F stats that are exactly 0 (that's only possible if the sample means are all exactly equal), so you could try to fiddle with how the F values are rounded. It's not great for them to show up as zero in the tooltips.

ethansaxenian commented 3 years ago

Good! At least it's working in the sense of all red dots to the right. Don't forget to fix the F stats in the top part of the module as well.

Thanks for the reminder!

As far as the scaling goes, there shouldn't be any F stats that are exactly 0 (that's only possible if the sample means are all exactly equal), so you could try to fiddle with how the F values are rounded. It's not great for them to show up as zero in the tooltips.

It looks like there are always some points that end up 0, even with no rounding. Not sure what to do here.

ProfessorAmanda commented 3 years ago

It looks like there are always some points that end up 0, even with no rounding. Not sure what to do here.

Interesting. I mean, it's possible, just unlikely. If you could print for yourself a report of the sample means and the overall means that are generated by the replications, you can check to make sure the math is right. F=0 if and only if all sample means are equal.

ethansaxenian commented 3 years ago

Interesting. I mean, it's possible, just unlikely. If you could print for yourself a report of the sample means and the overall means that are generated by the replications, you can check to make sure the math is right. F=0 if and only if all sample means are equal.

Actually, this makes a lot of sense. I've only been testing with 2 populations (the default), and the population observations are all rounded to the nearest whole number, so it does happen that sample means are equal.

ProfessorAmanda commented 3 years ago

and the population observations are all rounded to the nearest whole number

I get why it makes sense to round to the nearest whole number for the pictures, but is it possible to use unrounded numbers to calculate the SSTR, SSE, and F stats? Or numbers rounded to the nearest .1 or .01?

ProfessorAmanda commented 3 years ago

(We are a little bit splitting hairs, since it only seems to really matter for 2 populations)

ethansaxenian commented 3 years ago

(We are a little bit splitting hairs, since it only seems to really matter for 2 populations)

This is true. Regardless, I've just pushed a change that rounds the populations to the nearest 0.1 (for the pictures too) and it solves the problem, although the pictures look a bit weirder. Let me know what you think.

ProfessorAmanda commented 3 years ago

I like it! Pictures look okay to me.

ethansaxenian commented 3 years ago

I like it! Pictures look okay to me.

I actually like it a lot less. I say we either live with it as is or, since it only matters with 2 populations, limit to a minimum of 3

ProfessorAmanda commented 3 years ago

Maybe I should ask: which pictures look odd to you? The samples, maybe?

ProfessorAmanda commented 3 years ago

Found another math error in the portion above, which may apply to the type 1 simulation as well.

Check the calculation of the MSE. I did one by hand, and it was off by a factor of about ten. The MSE should be the SSE / (total sample size - number of groups), i.e. it should be the SSE / the degrees of freedom in the "due to error" row.

ProfessorAmanda commented 3 years ago

Looks killer! See you in ten.

ProfessorAmanda commented 3 years ago

Add to the distribution title, in parentheses, (df_num = [number of groups - 1], df_den = [total sample size - num of groups])

ProfessorAmanda commented 3 years ago

Looks good! Small mistake: df_num = number of groups - 1 (i.e. don't forget to subtract 1)