Homework and Tutorial 4

[lucieyang1]

~I'm still working on the tutorial, but the homework is done.~

• The homework has a lot of optional work, which I kept, but it might be worth seeing if some of it can somehow be integrated into the tutorial
• For Q16, I could not figure out how to make the sampling distribution dotplot with plotly, so I currently just have a histogram.
• I am also having some trouble creating the automated tests on MarkUs; when I click save, it just says "Queued" but my test disappears...

For the tutorial, I'm not too sure what material to cover. Last year's tutorial seems to be mostly focused on the distinctions with hypothesis testing and type I/II errors, but Week 4 will now come before those concepts are covered. I was thinking of using some of the examples from last year's lectures on how well samples approximate populations and getting tighter confidence intervals. Some guidance here would be appreciated!

Edit: I uploaded a draft of the tutorial, but I have around 20-30 minutes I'd like to fill with some practice or discussion. I'm not sure what to cover here.

[pointOfive]

My initial reactions to your helpful comments above

• Optional to Tut: I'm keeping this in mind as I look things over!
• Q16 -- will you look at Matthew's week 5 notebooks? One of those uses a dot plot... I think it's the Fisher notebook, if I remember correctly?
• I will review the MarkUs autotests... this stuff can be super finicky... I should be able to figure out what's happening and let you know (not super important a this stage, but just to dispel any lingering mystery).
• Regarding Tut: "I was thinking of using some of the examples from last year's lectures on how well samples approximate populations and getting tighter confidence intervals. Some guidance here would be appreciated!" Yes... this seems exactly like what we should do... explore the behaviour of sample size... exactly the kind of material I want to use tut for!
  • I'd like this to be made into live runnable code that can be experimented with during tutorial... do you have time to do this?
  • I think this might be what we could use to fill in the extra time; but, we'll see as I have a closer look at what's present

HW

• Q0: great hint... this makes me think that I'd maybe somewhere like to talk about what an observation is versus a measurements... I think this might belong with Rohan's week 8...
• Q1/Q2: I'm editing; but, I generally think this is all super great so far
  • my edits are all about providing a little more guidance and explanation: it seems I basically want my homework to be an interactive reading assignment where a concept is explained, and then used
  • I converted Q2 to a multiple choice question... I want to automate feedback as much as possible
• Q3/4/5/6: made some changes/edits as usual (per my sensibilities that arise as I review this kind of material); but, generally, I very much like what's going on in this homework.
• I appreciate very much that this material is generally quite close to "finalized"
• Q7/8: I'm liking a return to shape considerations here... seems like it could be usefully leveraged into "sampling distribution shape is effected by data distribribution and narrows towards normality with increasing sample sizes..."
• Q9: made this an autofail to provide more code feedback to students
• Q10: please convert this test to the way Q4 is tested
  • It looks like this wrongly reused Q3
• Q11: I like bringing the population back in (and sampling from that...)... so I'm going to make these optional "more auto claims data" questions actually required
  • I've made Q11 a problem that is now "visually checked by a TA; so, no printout will be made here"
  • I'm just thinking that by now there's been two problems with opportunities for feedback to see how to make these kinds of things; and, although this is asking for "samples from a population" as opposed to "bootstrap samples from a sample", I think students should be able to do this.
  • Actually... I'm going to add in a multiple choice question here to further clarify the point that we're trying to make here
  • Q11: please make the ABCD test cell for this
  • Note that I'm changing this to median (from mean) to match the analysis on the sample
• New Q12: repeat Q7 and Q8 but now for the population
  • Old Q12 is converted into these two questions above
• New Q13: create a multiple choice question with compound options as to weather or not the distribution observed in the sample seems to approximately matches the distribution of the population and if the bootstrap confidence interval based on the sample captured the true population average
  • Old Q13 is to be removed -- it's too redundant at this point
• **New Q14: set up a (relatively complex) problem and autotesting framework in which the students create 80% confidence intervals for the 1000 samples in Q11; report how many of the 1000 samples captured to the population median; and confirm this (probably by having the students submit the the samples and then recomputing and confirming their reported number of 80% confidence intervals that contained the true population median
  • Old Q14 is to be removed -- I don't think it approaches this materials as effectively as the revisions I'm requesting above
• New Q15: ask a free response or ABCD question that gets the students to consider that the original sample is based on an age cutoff, and so it's maybe not as representative as a random sample might be
  • it should somehow as well be emphasized that it's not really a "random" sample the way these new random samples are... so it doesn't quite have the "80% correct" "guarantee" that comes from assuming a random sample... would like to get this point emphasized as well...
  • I haven't checked, but perhaps an 80% confidence interval for the original sample won't contain the true population median? If this is so this could be emphasized throughout this current development? No worries if not the idea here can still be gotten across in other ways, I think.

I currently agree that "old 15-20" should likely be canibalized into the tutorial.

[pointOfive]

Tut

• first few slides are super great (very minor edits from me)
• Added an slide around how to "say/use" confidence intervals (language)
  • Maybe add a slide with a funny "finger waggling" grammar police image and a funny broken heart image (as I've made a "love" joke)?
  • If you can format everything to be on the same slide that'd be great, but isn't necessary :)
  • I've added a slide suggesting a possible suggested follow up to this: would you have a look and consider if it makes sense?
• "Full Class Discussion" slides currently images from my last year slides, following the "Confidence Interval Widths"
  • this would be great material for the tutorial... yes, exactly perfect and correct plan here
  • Can you make this live code that the TAs can demo and play with?
• I like the quiz: I've edited just a bit
• Please have consider how the time is looking. I'd like to target 100 minutes total... is this possible with all the suggestions and material this Tut now has?
• I merged the quiz slides and made instruction for reviewing... it's a little hacky, but TAs should figure out how to highlight the question and corresponding answer to talk through it
• Re: "Practice/Discussion (20-30 mins) - not sure what to put here" I've sort of given an idea; and, hopefully there's time to support this (after creating and including the demo code that the TAs will show above)?
  • See if you have any other relevant ideas for a question here
• Tutorial assignment: Will you look at the Week 5 and Week 7 and Week 9 Tutorial prompts (from the other guys) -- I feel like the formatting here could be a little more "polished" perhaps? Using the template that I've sort of gotten going in those other tutorials? It may actually indeed be the case that you're already doing this... I'm not sure... it just feels a little rough drafty to me at the moment...
  • Regardless/Nonetheless... I'd like the prompt to be more specific. Can you craft a prompt which asks students to describe how to estimate what proportion of the candy bought for halloween is their favourite candy based on (a) a bag of candy they get when tricker treating, and then to compare and contrast the bootstrap sampling distribution used to (b) the sampling distribution of the proportion of their favourite candy created from the proportions observed in each bag of candy for each student in the class? [and you'll need to reword and craft this a little more carefully to make it a good question; but, hopefully what I've started here gives you enough of the idea./?]
  • I'll let you decide; but, I feel like we can dispense with the vocabulary terms listing? Not sure, maybe not; but, I would think students should be using these terms to explain things... perhaps we can just drop the requirement that they "use two terms" or whatever that requirement is?

[lucieyang1]

I uploaded my edits to HW4. I haven't yet made the changes to the student-facing file, but will do so once the tester changes are finalized.

For Q15, I currently have it asking about constructing an 80% confidence interval, but I was thinking of changing it to refer to the 95% confidence interval since that was what was calculated earlier in Q10, although I don't think it's that important. It seems like the original sample is actually pretty representative of the population, and that both the 80% and 95% confidence intervals from the original sample do capture the true population median.

[lucieyang1]

I uploaded my edits to the tutorial now. I moved around the order a bit and now the content should take around 80 minutes, leaving 20 minutes for working on the tutorial assignment.

For one of the code demos, I put the import statements and made a helper function on a slide to be skipped so that the relevant code and plot on the next slide can fit, however, I'm not sure if it will still run when presenting the slides? Or if there's any better way to do this.

[pointOfive]

HW tester

• Q0-10: some edits; but, generally all good and just preferences/corrections stuff that always comes up in my reviews :)
• Q11-16: ended up feeling a different ordering and emphasis in the question prompts was needed here to create the effect I was hoping for; so, have done so!
• New Q17: this one wasn't quite what I was looking for (although I might not have described what I was looking for well) as all it really does is check if np.quantile does what it claims.
  • "fixed" it to be more like what I was looking for.

Can you see if you can get a student facing version of this file working?

[lucieyang1]

• For your edits of the Q1 test, it doesn't seem like it is actually testing left_count of the roaddata_sample. I changed it back so that it will test it, let me know if there was a reason for the change or if I should change it back.
  • I did notice that left_count = roaddata.sample(n=100)['road_side'].value_counts()['left'] oddly gives a different number each time, while doing road_sample = roaddata.sample(n=100) and roaddata_sample['road_side'].value_counts()['left'] separately seems to give 30, even with the same seed.
• I'm not sure if I'm misunderstanding Q17, but I can't seem to get the test answer from the template given (assuming I'm not supposed to add in more for loops). It seems like a separate loop, for one_sample in all_samples is needed before the for j in range(number_of_bootstrap_samples_per_sample) loop, though I'm not sure why it doesn't work without it...

[pointOfive]

Great catches -- wonderful care and attention to detail -- really good awareness.

• Q1: Oops -- looks like my assert left_count != 30 was wrong and it should be assert left_count == 30 like you fixed it to be. Sorry! (I must have wanted to see the test fail and then forgot to fix it back...)
  • I think the behaviour, re: left_count = roaddata.sample(n=100)['road_side'].value_counts()['left'] versus road_sample = roaddata.sample(n=100); roaddata_sample['road_side'].value_counts()['left'] is only because np.random.seed(130) only needs to be called once to make the second version always be the same; whereas, the version I used needs to always set the random seed as np.random.seed(130); left_count = roaddata.sample(n=100)['road_side'].value_counts()['left'].
  • I'll be careful when I review to make sure this is working
• Q17: there's a similar problem (which I myself initially accidentally overlooked and introduced), which is that the use of the random numbers is actually different in the solution loop because in template loop uses the random seed to produce samples as well as bootstrap; so, in the solution code if you just add np.random.choice(population['PAID'], sample_size_n, replace=False) (with no assignment) as the first line in the first for loop, then the random numbers will be getting used the same and will now match... do you see what I mean?