Homework and Tutorial 5

[MatthewYu06]

Thanks for waiting, I included sample solutions in the homework labelled "SAMPLE SOLUTION" I had some concerns while making the homework and tutorial: Homework:

• Lot of optional practice, I kept it optional
• Didn't find an efficient way to make dotplots with matplotlib or plotly, ended up brute forcing an output so student can answer later questions about the dotplot. Would appreciate any advice on this

Tutorial:

• Might be a bit lengthy
• I kept the group presentation intro in this week, maybe move to next week as involve 2 sample hypothesis testing
• Not 100% with validity of worked example, especially for median/wilcoxon test where I searched some things up

[Bortoise]

Overall the homework took me around 2 hours (including the optional parts). I have a couple of comments

1. p is not defined in the warm up, it's clear from context that p is the proportion of students who use twitter, however it might be worthwhile to say so explicitly.
2. You should be able to have the autograding handle Q4 pretty easily.

I like the first set of optional exercises, but they make the main part of the homework feel repetitive and samey (which is not necessarily a bad thing, just something to keep in mind).

[pointOfive]

Okay -- looks like my previous comment doesn't exactly have any serious conflicts with my newly updated thoughts on things; so, I'll start pretty fresh here about what I'm thinking (having not yet had any sort of look at any revisions you may have made to the HW).

• Fisher content was great/inspiring and so has been reworked into: Fisher_p_values.ipynb
  • But, rather than being a test/midterm question, I want this to be the "Tutorial Assignment"
  • "Tutorial Assignments" should be 1 hour or so activities which ideally can be started (but perhaps not finished) in the tutorial period; and, usually, these assignments should be "communication" focused.
  • This tutorial assignment, however, will be more like a regular homework assignment; although, it has a lot of "write answers" kind of questions; so, I think it is sufficiently "communication" oriented
  • Since the approach to p-value stuff this time is "expanded" to include binomial sign tests and t-tests I want to devote more practice time to these topics; and, so, I think another practice round of this material is reasonably in order
  • Additionally, I'll be demonstrating p-value calculations in class using normally distributed data as opposed to coin flipping data; so, the tutorial period is a review of all of this stuff again, but this time with coin flipping style data; so, again, having another pass to see the generalization ideas as well as variations on problem specifics is a good thing in my view
  • We'll need a tester file for Fisher_p_values.ipynb, then; and, we'll need example answers for all the questions that are automatically failed tests so that they show up as feedback for the student who review answers through MarkUs.
  • We can budget 2-3 minutes of TA time to open up these files and review a couple answers: I've proposed a couple, but would you think about which answers would be best to have the TAs review?
  • I eventually gave up on editing Fisher_p_values.ipynb so that you could put some fresh eyes on it and catch anything weird in there that I'm missing... please be openminded in your review of that file in the event that you see further opportunities for improvements in the exercise that might better student experience with this notebook.

I've also made some decisions about what we should do with the tutorial for week 5.

• Broke apart McStat problem as this needs to format and show nicely in present mode (know what I mean?)
  • view -> cell toolbar -> slideshow -> then mark slides and subsides etc as I have done
  • then you can hit the little graph button to the right of the "Viola" button on the top to go into "presentation mode"
• I started revising the outline (which I think was/is a good touch), but this needs to be given a little more attention/finalized
• I've worked on the discussion/code presentation as well and gotten that into a format I like through "Code III"
  • Please add to the tutorial a couple more code sections in this style which do the same sort of demonstration with the theoretical binomial distribution and t-distribution/t-test (and their p-values) that is done near the end of the Fisher_p_values.ipynb: we don't want this to take a lot of time for the TA; but, we want to touch on it so that it's not too weird in the students homework exercise
  • I'm now thinking we could drop this kind of "binomial / t-test" material from the homework (even though I had initially written you a note to include something like this in one of the later homework problems) in favour of having it be covered a little bit this way in tutorial and in the tutorial assignment
• I made the initial activity a group activity which will be the "Tutorial Activity".
  • "Tutorial Activities" are basically roll call / participation / engagement scores: something that the TA can quickly give a mark for as long as the student engaged in the activity; so, we need something like that for week 5; and, it seemed like the group exercise of talking and thinking about Stella McStat will fit the bill just fine.
• We'll drop the Type I/II error discussion from the tutorial: this gets covered well/fine/good enough in the homework as specified with the revision now
• Now, save all the content you made somewhere, as I think it will have a place to go eventually; but, basically everything after the "evidence against the null" table is currently "scratch work / ongoing draft" material
  • All the stuff about non-simulation based p-values should be streamlined/reworked into the way I present these topics at the end of the Fisher_p_values.ipynb. I'm not saying things here are "wrong" (as I honestly haven't looked at this very closely at this point); instead, just that we've gotta be very streamlined with showing the theoretical binomial distribution for this problem and producing a p-value from it, and then showing the distribution that the t-test uses to compute its p-value. Wherever you think your material helps/supports this then let's put it in; but, the point of having all of this is to show repeat the "as or more extreme" p-value exercise; and, show that different kinds of p-values exist. It's possible the exposition you've created so far indeed has the correct home in the tutorial slides; or, perhaps it might fit nicely into Fisher_p_values.ipynb as additional helpful explanation information there.
  • Your "discussion of the three tests" is a very good idea in my view, if there's time... I'll let you think about all the timing of things... you'll see that I've given my estimates up to "Code III" but you may revise them if you feel it's appropriate to do so
  • Let's plan to move the currently proposed tutorial activity into the next week, as students won't yet know what two-sample versus one-sample testing is at this point
  • Once you think you've settled everything down to a near finalized state, let's see if it makes sense to have a vocab listing in this tutorial... perhaps it could be more effective to move it to the next tutorial when two-sample testing comes along and basically everything a review but just with two-sample possibilities now as well.
• Also, whatever tutorial you had planned for week 6 will potentially get moved to week 7; so, as you think about organizing content between weeks, consider that you have the ability to have a third tutorial in which we could do hypothesis testing stuff
  • That would be the tutorial attached to the class in which we introduce the course project; so, getting students thinking about hypothesis testing stuff that they could do on class project data during tutorial time would be a great activity.

[MatthewYu06]

Ready for Review I'll try to make assignments on MarkUs soon to test out autotesting

Currently worked on files:

• Homework: Homework_5_student_4.ipynb/Homework_5_tester_4.ipynb
• Tutorial Assignment: Fisher_p_values.ipynb/Fisher_p_values_student.ipynb
• Tutorial Slides: Tutorial_5_draft_2.ipynb

Homework:

• Currently have autotests for written questions as print statements, perhaps better way?

Tutorial Slides:

• I think the theoretical binomial distribution and t-test does a good job at the "as or more extreme" excercise and showing different p-values exist, moved my discussion of my 3 tests to brief bonus questions at the end of Fisher_p_values.ipynb in case student wants more exposure
• I think a vocab list would make sense in week 6 tutorial as a "total summary" for both types of hypothesis testing

Tutorial Assignment:

• added Q13 and Q14 as brief introductions to more types of statistical tests and p-value calculation, feel free to edit if anything is wrong or remove these questions if you feel they're unnecessary

[pointOfive]

Homework and tester were in good shape as far as I could tell (with the exception of Q13 which needed a good amount of work in my view). My review and edits are below (please review/confirm, Matthew); and, they were mostly pretty easy to do with everything here mostly already in pretty good shape at this point in time.

• Please see the tester file for how hints can be added as test output: Q0/Q1/Q7
  • See Q5/Q6 for how tests can be "hidden" so they don't show in the output. I didn't feel the need to do this for Q0/Q1/Q7 (where the hints function more as an "explanation"); but, it seemed better to actually make the hint a hint or a key informative/explanatory statement rather than giving away the answer for Q5/Q6, etc.
• Q2 answer entry format can be more straight forward: look how I did this
  • I don't know for sure if/how MarkUs handles multiple assert statements; so, I've made this an "and" composite test
• Q3 reworked to have the auto format I'd prefer; also, be careful about overwriting variable names like simulated_sample_n400 as this would lose the student information
  • "Due to the randomness of seeds, your graph and response may look slightly different" is not right: students should get the same results using the same random number seed
  • Your written explanation/hint/example solution was otherwise very nice/good
• Q4: good job with this; so, I just edited slightly per personal sensibilities
  • Do notice how this is MarkUs output formatted for future reference
• Q7 needed an update (from my previous version) for correctness
• Q8 very nice: added my preferred prompting/formatting to request specific rounding choices
  • Formatted tester file to again preferred output styling
  • Notice how "Your p-value should be "+str(p_value)+" according to the code below; but, " can be used to expose answer information that might help explain to students what exactly is going wrong with their answer (so they can troubleshoot it for us, especially if something's not working on our end); and, here, of course, we do actually need to run the code (so we have these answers) as well as formatting it for output
• Q9 answer hidden and hint statement provided
• Q10 -- I like the way you did this very much: I only reformatted slightly, using your code essentially just as you used it; and, in this case, I thought it was okay to put everything into the same test cell, which means it will all be exposed to the students.
• We'll make Q12 a check MarkUs autotests problem to save some TA time
• Q13 should also use Q4 random seed: I've corrected this but would prefer for you to keep this kind of consistency in the future; and, there are only 4 digits of accuracy possible here; and, assert statement based on assert p_value == test_p_value is not going to work as it should be Q13 (although Q14 "worked" because you did there assign p_value a value)
  • I'm really hoping to not have to catch these sorts of problems in the "finalized" materials that you guys create and provide for me.
  • I've changed around the naming a little bit: I don't think p is the right choice as this refers to a proportion, but you've not structured your answer in terms of proportions (which I actually think is quite nice/good, btw).
  • I don't mind this approach to exposing the code here in Q13: it's a little simpler than adding a cell above which executes the code and then formatting the code as text output the way I do for Q3/Q4. I think the difference for me is that we need to run this code to get/check the answer, so it's thus efficient to just have the code run live (and get shown by MarkUs) in the tester cell.

Didn't catch anything new that needed to be addressed in the remainder of the questions Q15-19: hopefully indeed they're okay at this point

Going to try these out in MarkUs now :)

[pointOfive]

I needed to make a few edits.

• fixed Q2 testing solution
• the provided Q4 solution was actually based on the wrong problem... definitely want things cleaner than this moving forward
• Q8 has three digits of significance since it's based on simulated_samples,n = 1000: fixed this up
• Fixed a mixup I added between Q11/Q12

[pointOfive]

Caught a couple more errors, e.g.,

• Q13: latex needs to be double backslashed: e.g, \\text instead of \text
• I "fixed" Q2 wrong -- so that's now corrected

[pointOfive]

TUT

• flushed out the outline to be more informative/useful
• made smallish edits of the kind that strike my fancy as I re-review things
• I thought things looked in pretty good shape here I need to edit the last slide of the TUT for instructions on the fisher notebook: I'm going to wait to do that once that notebook is finalized for my last review/finalization.

[MatthewYu06]

Homework:

• read through to get feel of expected format
• slightly modified some comments in sample answers to be more clear (Q2, Q10)
• typo fix Q12

[MatthewYu06]

Fisher:

• modified solutions
• added quick hint but kept Q5, Q6 answers unhidden
• Q13, Q14 still necessary or too much?

[pointOfive]

I've gone ahead and removed Q13/Q14 -- contents there are still available in commit history; so/and, it could be repurposes for the tutorial next week.

Everything else looked in good shape with the edits I've made just being the usual kinds of things I see upon re-review.

Very pleased with the fisher homework for the tutorial.

[pointOfive]

So now I just need to update the tutorial assignment to point to the fisher exercise... will do tomorrow first thing I think.

[pointOfive]

And now we're just waiting for one more review from @mistryrohan of the Homework; and, let's also have Rohan review the Fisher notebook as well. And that should finalize everything here!

[MatthewYu06]

@mistryrohan if the folders aren't sorted by the time you are ready to review, the files to review are: Homework: Homework_5_tester_4.ipynb, Homework_5_student_4.ipynb Tutorial: Tutorial_5_draft_2.ipynb Fisher: Fisher_p_values.ipynb, Fisher_p_values_student.ipynb

Otherwise just review whats there

[pointOfive]

Oops -- did not describe that right:

• Tutorial_5_draft_2.ipynb -> STA130_TUT_5.ipynb, yes; but, then,
• Tutorial_5.ipynb -> Tutorial_5_old.ipynb

[mistryrohan]

As per Slack messages I have sent all the edited files to Matthew, do not know if there are issues with now my edits + Scott's new commits. If that is the case, I am happy to go over the files once more and make the same changes.

[pointOfive]

Hey @MatthewYu06 -- there is not Homework_5_student_4.ipynb; instead, you've renamed it Homework_5_student_4._ipynb.

Actually, this is not the way we should be doing this. You want to make edits on the same filename so that all changes can be observed and tracked. When you just delete Homework_5_student_4.ipynb and (then make some changes?) and replace it with the different file Homework_5_student_4._ipynb all change tracking done by git IS LOST!

Do you see what I mean?

Edit files directly. File renaming should be done itself with git mv file_old.name file_new.name and is it self a new commit. I am demonstrating this with my last few commits.

[pointOfive]

Hey @mistryrohan and @MatthewYu06 -- it sounds like edits are not being made as commits directly into the PR as I have been doing. This means we'll have potential conflicts.

• @mistryrohan: you have not figured out the public key situation yet so that you can directly commit to PRs?
• @MatthewYu06 have you adopted suggested changes from @mistryrohan relative to my own edits? If so, how was this done? Are these part of the newly named files with no edit history tracking?

[pointOfive]

@mistryrohan I'm not sure what slack messages you're referring to: DMs with Matthew? I thought our messages were about figuring out how to make direct commits to PRs?

• When were these edits sent to @MatthewYu06? Are they a part of his Homework_5_student_4._ipynb new file creation?

[pointOfive]

Final final naming update for @mistryrohan re: @MatthewYu06 (https://github.com/pointOfive/STA130_F23/pull/16#issuecomment-1614014783, etc.)

• Homework_5_student4.ipynb -> STA130_HW_5.ipynb
• Homework_5_tester_4.ipynb -> STA130_HW_5_tester.ipynb
• Tutorial_5_draft_2.ipynb -> STA130_TUT_5.ipynb
• Fisher_p_values_student.ipynb as is
• Fisher_p_values.ipynb -> Fisher_p_values_tester.ipynb

[mistryrohan]

@pointOfive These were sent before your renaming of the files. Also, I am refering to Slack direct messages from last night in response to yours (sorry for not specifying). I got the ssh key onto my account but still cannot clone it. Commands like clip, clip.exe, others from your instructions, and more do not seem to work. Cloning still asks for a public key (even though it is there now) and it was taking hours figuring it out so to not create more backlog I just sent changes to Matthew to commit on my behalf.

Edit: If the only changes now were just file renaming there should be no conflicts when Matthew pushes my changes

[pointOfive]

Oh sorry -- it seems slack was not delivering me messages... hmm... let me have a look at things and catch up! Will move the conversation over there as it might be a little smoother than here if it's working again.