Outrageous suggestion, dance of the p values

[gdcumming]

Dance of the p values..

The most compelling version of 'dance of the p values' uses the difference between two independent groups, so there are two pop curves, e.g. for Control and Experimental populations. But the dance is just as insane for a single group design--in CIjumping we'd need to be able to specify an H0 value that's different from the pop mean, mu. Then display the p values for each sample in a column to the left side of the RH panel. The pic below shows the general idea. Could this display be added as option in CIjumping?

Even better if there is sound, so deep blue comes with a deep note (trombone).... through to strong red, which comes with a high note (trumpet). (I can send the 5 small sound files if that helps.) Vary N, and/or the difference between H0 and mu, and of course see changes in the proportions of *** and p>.10 results, but always crazy bouncing around of p values.

(I don't know how we see that pic, other than by pasting the url into a new tab.The pic is also in the wiki file in dropbox.)

[gfmoore]

It's next on the list AFTER sorting out the tablet view.

[gdcumming]

Gordon,

What a blizzard of goodies! Thank you. Fully 2,200 lines of code--I salute your skill and persistence, in the great cause of helping folks appreciate intuitively the extent of sampling variability.

I can report that Bob ran a highly successful 1.5 hour zoom workshop the other day, as part of the SIPS (Society for Improving Psychology Science) annual meeting. It was around 5am for me, but better times for most other folks who zoomed in--and they did indeed come from all over. (A couple of Glaswegian accents...) Bob's slides are here: https://www.dropbox.com/s/gk7j8rkfzeh6rkl/New%20Stats%20Workshop%20-%20SIPS%20-%202020.pptx?dl=0 [https://www.dropbox.com/static/images/spectrum-icons/generated/content/content-ppt-large.png]https://www.dropbox.com/s/gk7j8rkfzeh6rkl/New%20Stats%20Workshop%20-%20SIPS%20-%202020.pptx?dl=0 New Stats Workshop - SIPS - 2020.pptxhttps://www.dropbox.com/s/gk7j8rkfzeh6rkl/New%20Stats%20Workshop%20-%20SIPS%20-%202020.pptx?dl=0 Shared with Dropbox www.dropbox.com Bob essentially made the case for estimation, then walked through several examples, using esci in jamovi. A great set of slides, which includes many links.

He was going flat out to get through around 75% of the planned content. A bit of time for folks to work a couple of the examples for themselves. The live chat was really buzzing all through.

The workshop was limited to those who had managed to secure one of the limited number of registration slots to SIPS; it had 103 live participants. He did find a moment near the end to mention your great new version of the dances software, designed to be across-platform and across-devices.

We really will have to get down to writing the second edition very soon. (I have preliminary updates of around half the chapters, but lots and lots still to do.)

Anyway, it's all extremely exciting, with more and more signs that folks are understanding that this really is the way of the future, here now. Your CIjumping is a vital part.

We should think about that name, 'CIjumping'. It's not bad, imho, but is there something better out there, perhaps including 'dances'?

It's going to take me a bit to work through all your latest. So it's good that you are giving yourself a breather. More from me soon.

Keep safe,

Geoff

---

Geoff Cumming, DPhil, Emeritus Professor, School of Psychology and Public Health, La Trobe University, Melbourne Campus, Victoria, Australia 3086 Email: g.cumming@latrobe.edu.aumailto:g.cumming@latrobe.edu.au Intro textbook: Introduction to The New Statistics: Estimation, Open Science, and Beyond www.thenewstatistics.comhttp://www.thenewstatistics.com First book: Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis ww.thenewstatistics.comhttp://www.thenewstatistics.com/ Own page: http://www.latrobe.edu.au/she/contact-us/staff/profile?uname=GDCumming ESCI (Exploratory Software for Confidence Intervals): www.thenewstatistics.comhttp://www.thenewstatistics.com/ Introduction to the New Statistics is the first statistics textbook to focus on Open Science and the New Statistics. Instructors can obtain a free desk copy at https://www.routledge.com/resources/deskcopy. Order on Amazon http://www.amazon.com/dp/1138825522 [1464879986726_ITNS]

---

From: Gordon Moore notifications@github.com Sent: Saturday, 27 June 2020 4:10 AM To: gfmoore/ESCI-JS ESCI-JS@noreply.github.com Cc: Geoff Cumming g.cumming@latrobe.edu.au; Author author@noreply.github.com Subject: Re: [gfmoore/ESCI-JS] Outrageous suggestion, dance of the p values (#15)

It's next on the list AFTER sorting out the tablet view.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/gfmoore/ESCI-JS/issues/15#issuecomment-650321682, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANFVLUEXUBY6UGQ6T37MR4LRYTQCHANCNFSM4OAF2IUA.

[gfmoore]

Okay, so I've implemented a first go at dancing with p-values ;)

I've muted the darkest blue, so that the value can show through. I have only done a p-value for sigma known (z) and based on the h0=mean. I have the code for t test ready, but I suspect you will want all sorts of options added ;) so let me know.

btw, I hope you don't mind my saying, but the sounds are truly ... are there any softer sounds! Or are you a jazz fan, perhaps Stan Kenton and City of Glass?

laughing my head off.

Oh I used p-value intervals of > 0.1, 0.05-0.1, 0.01-0.05, 0.005-0.01, <0.005 Feel free to change these.

I am now having a rest whilst you close everything! lol

btw was the above comment means to be public??

[gdcumming]

Great to see (and hear)! I'll need to do some concerted work on choices, controls, logic, etc. Shortly.

Sounds? A mate of mine recorded brief snippets, improvised on his electronic keyboard, for the first dance of the p vals video. He also did the single notes for the simulation. I'm more than happy to consider any replacement ideas, but they need to be brief, so they can be produced fast enough for the running simulation. I'll send you the sound files, tho' maybe you have been able to fish them out of ESCI intact.

BTW, for the first book, I needed to tweak ESCI so it would run on Mac. There's a different Mac set of workbooks still online. I went to enormous lengths to find a way to call and play brief sounds in Excel on Mac, but failed--even after paying some experts. No-one could come up with anything better than opening iTunes! Or one or two other just as clunky solutions. And this for a machine that prides itself on multimedia! That's one reason that for ITNS ESCI there's no sound.

I'll work further on a spec for p dancing.

[gfmoore]

Uhmm , Mac, did it/does it work. I don't have access to a Mac.

I'm not sure if there isn't a bit of a lag as well. It uses the html media api, but I think there are others out there?

Yes I grabbed the sound files from an old esci version..

You obviously didn't get the reference to Stan Kenton and why would you? This was my Dad's favourite Jazz album. My mum and anyone else who heard it thought it was absolutely awful. I think that's why he liked it. It's challenging listening, and I like off the wall stuff. :)

I'll have a think about what sounds, I have a keyboard. I do think it gets quite a reaction from the audience - wakes them up lol, so maybe have a select dropdown for different sound families!!!

[gdcumming]

0.3.44

Sorry, no, I didn't get the reference! I think it sounds better, more dramatic, when all 5 sounds are burbling away in a random mix, esp. with the trumpet (p<.001, ***) coming over the top every now and again. It's a demo that certainly grabs a the attention of an audience, and I'm sure the sounds are a big part. Way more striking that the low burp-burp-burp... that predominates when, as now, the null hypothesis is true.

Mac should not be an issue--that was an Excel-on-Mac problem with playing sounds that I had.

I've only just begun to think about how the display should look, and be labelled, to be most convincing. Not a trivial set of issues. More soon.

[gfmoore]

Oh I used p-value intervals of > 0.1, 0.05-0.1, 0.01-0.05, 0.005-0.01, <0.005 Feel free to change these.

I've now changed these to

Oh I used p-value intervals of > 0.1, 0.05-0.1, 0.01-0.05, 0.001-0.01, <0.001 Feel free to change these. is this correct?

[gdcumming]

Yes, spot on. As well as the different colours, from deep blue to bright red, the labels I use are: p>.10 ? for .05<p<.10

• for p<.05 (and >.01) ** p<.01 (and >.001) *** p<.001

Note that APA style (The American Psychological Association Publication Manual), which is a must for all psych students and researchers, doesn't use the leading zero for any quantity that is limited to (0, 1), or actually (-1, 1), like p values, probabilities, and correlations.

I'm not being super-careful about 'less than' or 'less than or equals', etc, which grates, I'm sure, with statisticians and mathematicians, but is common in psych and makes things simpler with students.

[gfmoore]

I'm not being super-careful about 'less than' or 'less than or equals', etc, which grates, I'm sure, with statisticians and mathematicians, but is common in psych and makes things simpler with students.

Sadly, I have to be, otherwise a p-value of = to something will not get dealt with! Anyway they should have been taught in school maths how to deal with inequalities, it's on the pre-16 curriculum!! (I have to defend maths teaching!! :)

[gfmoore]

Oh, I think I've got it now how you want! ;)

[gdcumming]

0.3.56

I like it!

Yep, of course equality needs to be considered and assigned to the tail, or not. (I did a combination of maths, maths stats, and psych in my first degree, so have seen all sides of such issues.) I had to dive into ESCI to find what I did. I required p<.001 for **, etc, so p=.05 did not earn a single star. But, on planet Earth, it really* doesn't matter.

I'm still thinking about layout so we can have a mu0 line, whose positioning determines the true effect size assumed in the population, and thus the statistical power. And the distribution of p values that we'll see.

At the left, the colour patches are ok. Could we give more space to the middle label column, enough to show '***' and 'p>.10' comfortably, then the p value as now.

Is the current font the largest that will fit and look ok? Maybe bold? To make the values more prominent and easier to read.

Let's have SCA when 'Dance of the p values' is clicked on, or off.

For 'p values' can we have italic p? And let's omit the hyphen. Thanks.

[gfmoore]

Ok, so added extra space, used a larger font, SCA implemented, italic p.

[gdcumming]

0.3.60

Thanks for progress. Let's leave for now. For the future, full functionality needs spec from me for showing H0 line, and displaying delta (i.e. Cohen's d in the population, i.e. the true effect size). Need to spec also the initial setup when 'Dance of the p values' clicked on.

Maybe best to have the 'Dance of the p values' checkbox inactive, for now. Can't turn it on.

[gfmoore]

Do you mean it won't turn on (it seems ok my side?) or you don't want it to turn on? I'd prefer to leave it on as I will be using it with my class soon.

Yes when you spec it out, then I'm happy to add Cohen's d, but not just yet :)

Unless there's any more error's let's close this and open a new ticket when you're ready.

[gdcumming]

0.3.64

Happy to leave as is, for now.

I've been thinking of the best and simplest way to get the full impact of the dance. I'll open a new issue when I'm there.

[gdcumming]

0.3.64

tl;dr?!

If you are thinking of using this with students (Good on you!), let's do it properly. I think (hope) it won't be very complicated.

For a start, change the options for N from ...10, 12, 15, 20... to ...10, 12, 14, 16, 18, 20... because we want to have N=16 to use here.

When Panel 9 is clicked open, see:

---

First line--as now Null hypothesis H0: mu0 = 50 Population effect size, Cohen's delta =

---

So the H0 value is always fixed at 50. Use Greek (lower case) letters for mu, delta. For H0 and mu0 use full-sized zero, because subscript would probably be too tiny to be salient. (Especially for the label at the bottom of the mu0 line in the graph, I think.)

Cohen's delta = (mu - mu0) / sigma

(mu and sigma are as set in Panel 1; all the dances assume Panel 1 describes the population assumed to be true)

When Panel 9 is clicked open we should get SCA, then see a default state, which is: Just as when ESCI-JS is originally downloaded, except: mu = 60 SD Lines ON N = 16 CIs ON mu0 line displayed--just like the mu line, but at 50 and with the label mu0 at the bottom. For colour, how about we try mu line blue (matches population) and mu0 line black? Both mu and mu0 labels at the bottom could be black.

With those default values we get delta = 0.5

When Panel 9 is clicked OFF we should get SCA and mu0 line disappears.

To calculate each p value:

Use two-tailed p values throughout

sigma known, use: z = (M - mu0) / (sigma / SQRT(N))

sigma unknown, use t with d.f. = (N - 1), and t = (M - mu0) / (s / SQRT(N))

--where M and s are the mean and SD of the current sample. (s should be calculated using (N - 1) as the divisor--most but not all software does that)

When using this dance, the mu0 line is always displayed and can't be turned off, but the mu line could be turned off.

---

The idea is that using a single group (as here) of N = 16 and delta = 0.5, the statistical power is close to .5, meaning that in the long run about 50% of samples will earn stars by having p < .05. One group, N = 16, is equivalent to (i.e. same power as) two groups each of N = 32, as I demo in the 'dance of the p values' videos. The distribution of p values depends on the power, and the power of around .5 in these two cases gives a good mix of p values to demo.

Change N and/or delta and get different distributions of p values. (To change delta, change mu and/or sigma.) For higher power (larger N, larger delta) expect more ** and *** p values, fewer p>.10. Lower power and v.v.

However (drum roll) whatever the power, whatever the N, delta, or anything else, p values will always dance wildly, just with different distributions.

One response I've had to the dances is to say, yes but we always knew that low power experiments are pretty useless. Two answers: (1) power of around .5 has been found to be typical in many fields of psychology research over the full period from 1960 to quite recently, and (2) sure, if you have very high power (typically N>1,000 or delta > 2 or 3) then you are highly likely to get ***. But with such very high power studies, the effect "passes the intra-ocular trauma test" (it hits you between the eyes) and we don't need fancy t calculations to tell us that almost certainly we have a non-zero effect.

Even with high power, if you run the dance and report p values to 8 decimal places, say, they still dance around surprisingly much, just all being small.

You may care to search YouTube for 'significance roulette' to find two videos that go beyond the dance of the p values. They explore the distribution of p values when we run an exact replication of a study, knowing only the p value given by the original study. Just like roulette, it turns out. Even more astonishing than dance of the p values, imho. Astonishing but true--there's reference in the videos to a 2008 article that explains.

In short, the defaults I suggest should give a good demo, and is an experiment that's fairly typical for psych, being equivalent to two independent groups of N=32 each.

I hope you find this bit, at least, fun!

[gfmoore]

When Panel 9 is clicked open we should get SCA, then see a default state, which is: Just as when ESCI-JS is originally downloaded, except: mu = 60 SD Lines ON N = 16 CIs ON mu0 line displayed--just like the mu line, but at 50 and with the label mu0 at the bottom. For colour, how about we try mu line blue (matches population) and mu0 line black? Both mu and mu0 labels at the bottom could be black.

With those default values we get delta = 0.5

done.

When Panel 9 is clicked OFF we should get SCA and mu0 line disappears.

done

When using this dance, the mu0 line is always displayed and can't be turned off, but the mu line could be turned off.

done

[gfmoore]

To calculate each p value:

Use two-tailed p values throughout

sigma known, use: z = (M - mu0) / (sigma / SQRT(N))

sigma unknown, use t with d.f. = (N - 1), and t = (M - mu0) / (s / SQRT(N))

--where M and s are the mean and SD of the current sample. (s should be calculated using (N - 1) as the divisor--most but not all software does that)

Ooops I had calculated the t, but forgot to use incorporate it, fixed.

[gfmoore]

When turn panel 9 (p value off), should CIs be turned off or left checked?

[gdcumming]

0.3.67

Lovely to see mu0 on screen! Getting close.

Some issues:

Pop SD lines are displayed. Change population shape, and SD lines don't come on with new shape, tho' SD lines checkbox is still ticked. (I don't think this is a Panel 9 issue.)

Click Panel 9 on, pop is not reset to normal, if it happens to have been something else. Also actually set mu=50, sigma=20.

Also need to turn on/off all the other checkboxes to correspond to defaults, plus amendments as in spec. I.e. can't assume Panel 2, 4 checkboxes are on, mu line is displayed, etc.

I think you are still calculating z and t with reference to mu, rather than mu0. (mu doesn't appear in the formulas, and is irrelevant to NHST; it can be turned off)

When Panel 9 clicked off, it's ok to leave CIs on, and other settings as they were. Just SCA.

[gfmoore]

I think you are still calculating z and t with reference to mu, rather than mu0. (mu doesn't appear in the formulas, and is irrelevant to NHST; it can be turned off)

Oops, yes. Schoolboy error. Forgot the 0 for mu0, so had mu, good catch.

Pop SD lines are displayed. Change population shape, and SD lines don't come on with new shape, tho' SD lines checkbox is still ticked. (I don't think this is a Panel 9 issue.)

So what's the difference between showSDLine = true and showSDline = true. Just a capital L!!! Should be sorted now.

Also need to turn on/off all the other checkboxes to correspond to defaults, plus amendments as in spec. I.e. can't assume Panel 2, 4 checkboxes are on, mu line is displayed, etc.

Sorry to be a pain, but you'll have to tell me again exactly what you want on or off? This could be a lot of work as I don't have a routine to set to defaults!!! The thought of going through all that again fills me with dread. It might be easy or it might not, so if you could just say what you want to see.

edit: I do have a stop, clearAll SC and that is called, but I assume you are thinking of resetting other checkboxes, values?

[gdcumming]

0.3.69

SD lines behave well.

mu0 fix, yes make a giant difference!

When click Panel 9 open: SCA Pop normal, mu=60, sigma=20 Panel 2: all 3 on Panel 4: N=16, all 3 on Panel 5: all 4 off (this is a change of mind from previous) Panel 6: C=95%, CIs on, sigma known Panel 7: mu line on, capture off Panel 8: closed

After Panel 9 is open, user can change lots of things (hide pop, turn on mean heap, etc, etc), which is fine. But a reset to all the above when Panel 9 is opened would be great. Hoping not too complicated!

[gfmoore]

So, I think I've done this. I'm sure you'll find something I missed. There may be some unintentional consequence that I didn't think about. As I said there are so many variables... Yes the dance is far more impressive now. :)

I also added a volume control for the sound. I note however that the sound doesn't always seem to get played. I am investigating. For instance set mu to 70 where all the p-values are very small and not every p-value generates a sound, though the code certainly hits that spot!

[gdcumming]

0.3.70

When this is all set up, it gives me enormous pleasure to see and hear it running! For me, too, this was close to the start of it all.

Yep, there is something weird with the sound. Even when sampling slowly (single clicks, or very slow Run) some sounds are missing, some are the wrong choice, sometimes we hear what sounds like two sounds superimposed. I think it happens whatever the value of mu. Some of this seems new--in previous versions sampling had to be slow to get one sound per sample, but I don't recall missing or wrong or mixed sounds when taking single or very slow Runs of samples. Strange.

On Panel 9 opening, many things seem correct :-)

But if I have, say, rectangular pop, mu=44, sigma=22, then opening Panel 9 doesn't reset to normal, 60, 20. If +/-MoE lines are on, the box is cleared but the lines are still visible.

[gfmoore]

But if I have, say, rectangular pop, mu=44, sigma=22, then opening Panel 9 doesn't reset to normal, 60, 20.

Didn't realise it had to do this, sorry.

If +/-MoE lines are on, the box is cleared but the lines are still visible.

Missed this as well, though I don't recall seeing it.

Not sure I'm even getting notifications since I renamed the github repository?

Okay, so can you review and let me know what I'm still missing please.

For your edification this is the code I had to do all the above. Perhaps you can understand why there are so many mistakes/bugs :(

//show the p-values $pvalue.on('change', function() { pvalue = $pvalue.is(':checked'); if (pvalue) { mu = 60; setMuSigmaSliderVal(mu, sigma); stop(); setOldMu();

  $normal.prop('checked', true);
  $rectangular.prop('checked', false);
  $skew.prop('checked', false);
  $custom.prop('checked', false);

  normal = true
  rectangular = false;
  skew = false;
  custom = false;
  changedDistribution = true;

  changedDistribution = true;

  //make sure these are on
  $showPopulationCurve.prop('checked', true);
  showPopulationCurve = true;

  $fillPopulation.prop('checked', true);    
  fillPopulation = true;

  $showSDLines.prop('checked', 'true');
  showSDLines = true;

  $showSamplePoints.prop('checked', true);
  showSamplePoints = true;

  $showSampleMeans.prop('checked', true);
  showSampleMeans = true;

  $dropSampleMeans.prop('checked', true);
  dropSampleMeans = true;

  $showMeanHeap.prop('checked', false);
  showMeanHeap = false;

  $showMeanHeapCurve.prop('checked', false);
  showMeanHeap = false;

  $showSELines.prop('checked', false);
  showSELines = false;

  $plusminusmoe.prop('checked', false);
  plusminusmoe = false;

  $ci.val('0.05');  //95% CI
  alpha = 0.05;

  $showMoe.prop('checked', true);
  showMoe = true;
  $showPmoe.prop('checked', true);
  showPmoe = true;
  showSmoe = false;

  $showCaptureMuLine.prop('checked', true);
  showCaptureMuLine = true;

  $captureOfMu.prop('checked', false);
  captureOfMu = false;

  $captureNextMean.prop('checked', false);
  $('#capturenextmeangrid').hide();
  captureNextMean = false;

  clearAll();

  n = 16;
  $('#samplesselected').val(n);

  $showMoe.prop('checked', 'true');
  showMoe = true;
  alpha = parseFloat($ci.val());
  // //I know mu and sigma so I know pmoe so redisplay it
  pmoe = jStat.normal.inv( 1-alpha/2.0, 0, 1 ) * sigma/Math.sqrt(n);
  if (showPopulationCurve) $pmoe.text(pmoe.toFixed(2));

  drawMu0Line();

  $('#pvaluesoundblock').show();
  $('#pvaluehypothesisblock').show();

  showpvalues();
  displayCohend();
}
else {
  $('#pvaluesoundblock').hide();
  $('#pvaluehypothesisblock').hide();

  stop();
  clearAll();

  $plusminusmoe.prop('checked', false);
  plusminusmoe = false;
  removePlusMinusMoe();

  hidepvalues();
  $pvaluesound.prop('checked', false);
  pvaluesound = false;
  clearCohend()
}

})

[gfmoore]

Okay, next job to look into the sound.

[gfmoore]

Okay so found a sound library called Howler that preloads the sounds into a cache. It seems much better, though the volume control I think is probably now logarithmic and the wrong way logarithmic) Will investigate further.

[gdcumming]

0.9.4 Beta

If +/-MoE lines are on, the box is cleared but the lines are still visible --When open Panel 9, pop is reset to normal 60 20, but MoE lines in position for previous pop are still visible--pls turn off

I'm suitably impressed, frightened, aghast, mystified.... by that code!

'Howler'?!!

Sounds: We still seem to get doubled up sounds a proportion of the time. I can't see a pattern of when it happens.

The and * symbols next to the colour patches look ok, but looks different--more like a bold dot? Is it practical to make just the , , * a bit larger, maybe bold, without changing the size of '?' in that column? Or maybe try everything in just that column as bold?

Would making the numerical values of p in the adjacent column bold make them harder or easier to read? Is it even possible/practical to do bold here?

[gfmoore]

Ahh, the +/- moe was removed if p value checkbox turned off, but not when p value moe turned on - rectified

My eyes are getting bad, I thought it was a bullet point, changed to *

Made things bold.

Geoff, mate, enough tweaking please! I could spend all my life tweaking ...

[gfmoore]

Ok, I think I need to make a personal decision here.

I am closing all issues now.

The software is going to have to be good enough.

I will fix any urgent bugs, but will not do any enhancements till September. Let's give it a rest and time to bed in.

If there are any new issues, then feel free to post them and perhaps label them as Urgent bug, or an other appropriate label.

I'm not a perfectionist. I'm an engineer! lol

Hope you are okay with that. Let's move on to other things.

If it's needed for the book, you'll have to photoshop it and I'll make changes in September, fair enough?

[gfmoore]

Sounds: We still seem to get doubled up sounds a proportion of the time. I can't see a pattern of when it happens.

I've looked at code, can't see anything. Sound on web pages is notoriously difficult, it's to do with caching and maybe there's a bug in Howler. I can't see anyway that the code can play more than one sound per p-value. If there was a doubling then we'd see it in the display, but hey I'll keep looking.

[gfmoore]

Bug fixed on duplicate sound because it was playing for sigma known and unknown. Only when there was a discrepancy did it sound a chord. Only found because I was trying out a different library and I got an error where I wasn't expecting it. You can look and look and just not see it!