Homework and Tutorial 7

[mistryrohan]

This is the "tester" version of both, I will make the student-facing one after changes from the review. Ignore the "Test" and "Testing changes" commits, those were for the video and I do not want to make a whole new branch just to pull my homework and tutorial.

Also, the tutorial images probably will not work here, but they're in the resources for TUT7, I just do not want the link to be local and people having to download it.

[mistryrohan]

I can fix the conflict above once someone edits the homework and tutorial files.

[pointOfive]

This is shaping up to be a great homework; but, I have quite a bit of revision that I'd like executed to get it there!

• Q0 added: I'd like to discuss missing values straight off the bat [as they should always be attended to straight away, I would say] (and then again after the log transform section)
  • I like the dropna() step in Q3, but it doesn't do anything for the newly created variables; so, please move this exercise right after the initial data import; and, change the unit test so that it checks the size of the data frame rather than the current unit test in Q3 which is passing even if I don't do any dropna() work. In Q3 it would be good at add a comment that log of 0 or negative numbers becomes a NaN and so have the students just double check that there's no created NaAs as a result of their created transformation.
• Q1: hint added
• new Q2 added: "add trend line"
• new Q3 added: calculate correlation
• new Q4 added which asks students to reflect on the correlation measure: TAs will not review this question as a sample answer should automatically be provided through MarkUs with a "test" that always fails to reveal this explanation. This will be a trick to use a lot to control the amount of time the TAs will need to devote to their manual reviews of homework problems.
• Now I think I'd like to here at this stage add a tutorial about the four assumptions of linear regression; specifically, I'd like to show the picture of normal distributions on a line (which is in my slides) [and/or google images "normal distributions on a line regression"] and point out heterscedasticity. It'll be worth pointing out the iid assumption here as well; but, that's more of an afterthought for us at the STA130 stage, tbh. So, you'll introduce/discuss the four assumptions of linear regression; then, we can have the students better critique this figure in terms of these assumptions
• Old Q2 becomes new Q5 where students critique normality/heterskedasticity relative to linearity (and independence is an after thought at this point and not the focus for us here yet honestly)
• OldQ3/NewQ6: edited slightly
• OldQ4/NewQ7: seems to have had a problem using the wrong variable (that I think carried over from the previous version of the homework) which I've corrected; so, answer changes with the corrected data; and, in the new description should be the notion that there's an artifact that can be seen in the upper right side of the trend line even though the single trend line appears kind of okay otherwise. The homoskedastic (constant) variance assumption appears better, but the the artifact in the upper right suggests that the normality assumption may still not be appropriate.
• New Q8: Make a multiple choice question here with the answer that the homoskedastic (constant) variance assumption has improved (but the normality assumption still appears questionable so that choice for what's improved wouldn't really be correct). Students should be comparing the plot with the transformed data versus the original scale data
• OldQ5/NewQ9: now they compute correlation again which would be expected to be better than before, but really it's not... as it's got "average" spread whereas the original data had tight spread on the left and loose spread on the right... somehow if we can also emphasize in this question that scaling the data (e.g., by 100k) does not affect correlation, that would be good to do
• New Q10: multiple choice answer that the correlation hasn't gotten much larger than it was in the untransformed data
• OldQ6/NewQ11: I want to make this more of a tutorial, walking the students through how to write this; so, perhaps first demonstrate how to do it with the "log(ave_tix)\~weekly_gross_100k" model then have them do it for the original "ave_tix\~weekly_gross" model
• OldQ7/NewQ12: same thing -- change this to an example/tutorial style followed by having them do it for the other model... so, tell/remind the students what the null hypothesis should be for one model and then getting them to write it out for the other
• OldQ8/NewQ13: same thing -- change this to an example/tutorial style by demonstrating it for one of the models followed by having them do it for the other model. Your guidance should also (of course) include the interpretation which discusses the caveats that the assumptions of the regression model do not appear to hold, so the statistical analysis may not be reliable; and, they should be doing the same in their version (where probably they're doing the log model).
• OldQ9/NewQ14: same thing -- I like this question coming in here a lot, but again, let's first demonstrate it for the students with one of the models and then have them do the same for the other model. At this stage, the R^2 only means to them "the proportion of variation explained"...
  • so (in conjunction with this question) [having not yet seen your tutorial slides] I'd like to add into the Friday tutorial a TA lecture on the meaning of the R^2. The tutorial lecture should reference this problem and return to the "log(ave_tix)\~weekly_gross_100k" and/or "ave_tix\~weekly_gross" models; and, it should show by example what is meant by "proportion of variation explained". So, the oldQ9/newQ14 topic (namely, R^2 as proportion of variation explained needs to discuss the sum of squared residuals AKA least squares fit) gets added to the tutorial. I've made a figure of this in my lecture slides from last year, so this figure and presentation now needs to be made into a lecture/conversation for tutorial.
• Original Q10/Q11: I like this very much. But now I want it more flushed out. Before jumping into the facet_plotting way to do this, let's look at the two models ave_tix\~weekly_gross (when Hamilton=="Yes") and ave_tix\~weekly_gross (when Hamilton=="No") and then go through the same exercise as above with the assumptions being better. So, they're doing the same thing but just with the two subsets of the data (making/fitting two trend line plots)
  • Demonstrate adding adding titles to reflect the data subsetting
• So there needs to be a question or two about if they'd like to fit a single model like in Q4 or fit two separate trend lines for the Hamilton data and non-hamilton data separately (including the subsequent trustworthiness of the associated statistical analyses).
• THEN move into your example where it can all be fit together as you show with facet_plotting stuff
• THEN we need a little more material showing that the facet model uses indicator variables (and what those are) [which I think is essentially what you're doing at the end]
  • indeed certainly when you all the data together a single straight line doesn't seem like it would be as good as two different lines, one for the Hamilton and one for the non-hamilton data, and we'd really like this conclusion to be emphasized come out clearly in this material (which I think it does and I like a lot)
• We want to use this wrap up to demonstrate more linear form specification using indicator variables and multiple linear regression specifications (to hint about the next topic)
• I'd also like you to consider adding a final final conclusion/demonstration/problem where you introduce the ave_tix\~weekly_grossHamilton + weekly_gross^2Hamilton model.
• I'd like to add/see a quadratic fit for the Hamilton data plot. A quickly look suggests this can't be done with plotly trend line; but, certainly it should be possible to fit and overlay such a fit on the hamilton data only plot. This could could just be supplied to the students. I'd then like a question asked to them about if they think the curved (quadratic) trend for the hamilton data looks preferable (with respect to the linear regression assumptions). The conclusion of the figures so far is that a linear line for the non-hamilton data looks pretty descent, with increased scattering on the right; the linear trend for log weekly sales for all data looks not too bad; except, there's a bar of points artifact caused by the hamilton data. When modelling the hamilton data itself, it seems a curved (quadratic) model is maybe interesting?
• [After looking at this data, the final model I think I'd use would take the form ave_tix~weekly_grossnon-Hamilton + weekly_grossHamilton + weekly_gross^2*Hamilton... not sure, but it seems like reintroducing this at the very end of this assignment just to show it would be a good lead in for the next week

[mistryrohan]

Should be good for a final small review if it is even needed. I added images to the 'im' folder in our GitHub (mimicking what Scott did for the syllabus so I hope it works).

[lucieyang1]

Do you have the assignment on MarkUs and the student-facing file? For testing. I don't see it on my end right now.

[mistryrohan]

I have added them, I am not sure if they are working as intended though. It does not show any test output on whether it passed or failed on my end after I made some changes.

Edit: Nevermind they do seem to work it was just really slow for some reason

[lucieyang1]

I see the MarkUs assignment now! The file in your fork seems to have solutions and the tests. Where can I find the student-facing file? Thanks :))

[mistryrohan]

I have to change that up since I had to do set seeds on the plots now so I should have that tomorrow before I review Mathew's new changes. Sorry about that.

[lucieyang1]

Hello, I don't think the MarkUs tests are working for the updated files, but I did compare my answers to the tester.

• Not sure if it's needed, but I think students might benefit from more MarkUs tests that they can get instant feedback on. I think you could make tests for Q0, Q3, Q6, and Q13 (testing beta_0 and beta_1 values?).
• For Q7, you have weekly_gross_overall_100k, when it should be weekly_gross_overall_in_100k to match with previous instructions
• For Q9 and Q14, rounding should probably be specified in the question
• For Q14, it looks like you used a rounded r (correlation) to determine R^2, so the value was off. I got 0.665 instead. Also, instead of squaring r, the R^2 is also given in the results.summary() as 0.665.
• Some of the written questions (Q12, Q14) in the student file still have "solution elements" in the answer.
• For Q16, it might be better to word the question as "compare advantages/disadvantages between faceting and colouring" or something to that effect, so that students look at them separately and provide 2 advantages and 2 disadvantages (1 for each of faceting and colouring). I wrongly assumed that I was looking for one advantage and disadvantage for both together.

Everything else looks good to me! :))

Edit: I have made all these edits directly to the files. I also changed the tester file so that students don't see the answer with MarkUs tests. Feel free to revert if you disagree with anything!! I also tried uploading the tester file on MarkUs and running tests, but they do not seem to be working. Not sure why.

[pointOfive]

Lucie and I ended up editing at the same time. Lucie's additions to the tester file won't conflict at all with anything I've done as I've not yet addressed the tester file.

For the student facing notebook, don't look at Lucie's edits until you've finalized my version updates (which was based on your last version). Then, click on Lucie's commits and see if something there could overwrite or improve what I've already provided in my commit.

[pointOfive]

Here are the notes I've made on my edits/updates commit:

• This assignment/exercise is going to be certified awesome: I’m really liking what’s coming together here.
• I’ve not yet gone over the tester file, and would like to have you(Rohan) first synch that up with everything here, and then I’ll do so
• You’ll need to remove my coded answers so this goes back to the blank student facing form it needs to be in.
  • This shouldn’t be too much work and should be pretty easy to do along with another pass through the notebook to make sure there’s no typos or anything small or large that needs to be tended to that I might have overlooked
  • One thing that would be great if you could do it is the bold all technical vocabulary terms. I’d like to do this with these notebooks and sometimes I am doing so but sometimes not…
• Anyway, please take time to review my edits as some of them are things that I think you could be able to correct prior to my review in the future.
• Definitely follow up on Lucie’s autotester requests, e.g., for Q3 (like is done for Q9): These are needed for sure!
• My proposed solution for Q3 is broadway[['weekly_gross_overall','avg_ticket_price']].corr().iloc[0,1].round(3), but you’ll see these around since I’ve left my work in the notebook for now.
  • Make sure all the MarkUs autotest feedback gives good hints and good answers and is generally helpful. Make sure to include correct code answers when appropriate
  • Please have a look at HW tester file in Lucie’s week 3 PR: this has the format that I’d like to output code answers with; and, also has the format that I’d like the MarkUs autotesting output to have for the “automatically failing” autotests that provide example answers that students can check out
• Q0 is going to be autotested based one .shape, right?
• Four assumptions presentation is outstanding, and has led to some synergistic and very productive edits and updates from me :)
• Q6 is autotested automatically, right? Based on broadway.shape?
• The Qx cells where answers are entered need to have more explicit instructions about the format of the required answers. I know it’s overhead and extremely tedious and redundant, and potentially even completely self-evident from your point of view; but, for the sake of consistency and having a super polished and professional product and experience for the students, I do want this to be done. E.g., Q3/Q10 need to reiterate exactly what should be assigned to these variables for testing purposes; and, the ABCD answers need to be the way they are in HW1/2/3
  • I’ve done this to the level of detail I’m looking for in some of the questions and the new Q15 I’m adding (as you’ll see)
  • Q14 needs help this way as well; and, autotesting needs to be very careful about decimal exactness when making comparison tests, which is why we need to be very explicit about what the expected answer format is
  • Please make sure all the Qx input cells are explicit about the requirements
• I’ve added a new Q11; so, unfortunately, please add in everything to process this and re-numerate the remaining problem numbers… sorry… I just think we need to ask this question (and I don’t want to do something like 11A/11B or something like that at this stage in the development
  • It’s worse than that though… when this cell correspondingly gets added to the tester file the cell id will need to be changed in the tester file to match the corresponding cell id in the student file.
  • #test_Qx cells only appear in the tester file and have their own id
  • The student facing file cells overwrite the cells in the tester file that have the same id as long as the ordering of the two files is the same
  • So, you’d add cells to the tester file corresponding to the cells I’ve added in the student facing file; then, change the ids of the cells (by opening the tester .ipynb as a text style file and finding the cell and editing the ids to match the student facing file)
  • C is the right answer for new Q11
• The old Q11 is probably the one we should have the TAs manually review, but you can consider this as well as you review the notebook. I’ve added a clear statement of how the TAs are to mark this assignment at the top of the notebook.
• Your conversation about hypothesis testing was good; but, your language felt to “causal” for me. I prefer a word like “associated” instead of “impacted”
• I’ve added a new problem after current Q13, breaking Q13 apart and thus listing it as Q15 so it lines up with the re-enumeration (beginning with my added Q11)
• Check out how you can add curved lines to a figure in my final additions to the notebook (where I changed the last problem)

[pointOfive]

Once you've given another pass based on my commits and then reviewed Lucie's commits, you'll finalize the tester file. It seem Lucie had some problems getting the tester file to work. Not sure, but that probably referred to student run tests since you thought they were working for you (probably for the instructor run tests). When you've got the final tester file all synchronized with the new way the homework is, see if you can get the tester file working. If not, let me know so I can take a look into what might be going wrong. These MarkUs tester files rarely seem to quite work right off the bat in my experience...

ps -- I've already said this, but Lucie's general comments/feedback about how submissions/testing needs to be specified just so are all exactly correct, so make sure the student facing file and tester file correctly address the issues Lucie's raised.

[pointOfive]

Tutorial commit incoming.

• added a code example demonstrating correlation calcs: this sort of thing is needed and should be added where possible. If we have to much stuff we'll take it out later; but, it could then be used for test questions, so it's definitely not wasted
• Markdown table formatting was not correct/working -- fixed
• Did some other revision formatting stuff -- just personal sensibilities as happens upon repeated reviews/revisions
• Looks like you created some code for the Starbucks example, but did not integrate this into a script in the TUT file (just a question of having time to get to this and noticing the request in the tut file, I suppose): I have done this now so you'll have an idea of what this should look like at this stage now
• changed the sequencing order of topics around just a bit
• only plotly figures, please. I do tend to like to show calculations with both numpy and pandas; but, for plots, let's just keep the world small with plotly
• There's too much content so I've specified times for each slide, including 0 minutes so the TAs will show, comment, and move on from these slides very quickly
  • In particular, I've removed the segment I created about OLS: I'd like to move this content into the next week; where, mostly it can be presented as is; but, the content should potentially also be incrementally extended to include the notion that this is how OLS fits extend into multiple linear regression
  • So notice how I use the "skip" option for the "sum of least squares" slides... I like doing this to keep this content in here available; but, make it now show up on the presentation version: I think that's how we should do this
  • Copy this material and lead with it for the next tutorial I think (at this point)
  • ... I think the "incremental improvement" would be to also make this plot "residual differences plot" for an interaction model as well, which would show how general OLS is; then, I think the rest of the tutorial for next week would be practicing interpreting interactions; and, maybe higher order terms for covariates in linear regression

[pointOfive]

Not sure where the disconnect was (perhaps just overall time demand), but it doesn't look like the things I asked for were completed. (i.e., mostly, tester file has not been synched/finalized!) I've gone ahead and done what's needed this time. Here are the notable edits I needed to make to address my requests above: https://github.com/pointOfive/STA130_F23/pull/15#issuecomment-1603050532

student facing (STA130_HW_7_student.ipynb)

• removed solutions from student file!!

tester (STA130_HW_7.ipynb)

• updated to match student facing files so there's no versioning differences that might be confusing later
• commented out broadway = broadway.drop('top_ticket_price', axis=1) since if this will thrown and error or make the answer automatically correct if run in the tester file!
• exposed some answers like plots Q1/Q2 in an automated way assigning hint in preceding cell (so it doesn't print twice) and then formatting the output in the way I'd like this to be done in the future
• Q3: hiding answer as test variable in preceding cell so it's not automatically exposed
• Solutions won't show in MarkUs unless they're output as messages with assert statements! [which is what I meant by my request to review how "autofails" are done in Lucie's HW3]
• synching versions and perhaps some small edits here and there
• Q11 answer missing an answer in the tester file? failed to update, it looks like

...

Oo... the tester file has not been updated to reflect the new indexing/enumerating of the student facing file... hmm...

Going to pause for a moment here and consider options. [Things are synched up to Q10/Q11 between STA130_HW_7.ipynb and STA130_HW_7_student.ipynb]

[pointOfive]

Ah -- I see on MarkUs STA130_HW_7_tester.ipynb from June 15: perhaps this is the updated file.

[mistryrohan]

Not sure where the disconnect was (perhaps just overall time demand), but it doesn't look like the things I asked for were completed. (i.e., mostly, tester file has not been synched/finalized!) I've gone ahead and done what's needed this time. Here are the notable edits I needed to make to address my requests above: #15 (comment)

student facing (STA130_HW_7_student.ipynb)

• removed solutions from student file!!

tester (STA130_HW_7.ipynb)

• updated to match student facing files so there's no versioning differences that might be confusing later
• commented out broadway = broadway.drop('top_ticket_price', axis=1) since if this will thrown and error or make the answer automatically correct if run in the tester file!
• exposed some answers like plots Q1/Q2 in an automated way assigning hint in preceding cell (so it doesn't print twice) and then formatting the output in the way I'd like this to be done in the future
• Q3: hiding answer as test variable in preceding cell so it's not automatically exposed
• Solutions won't show in MarkUs unless they're output as messages with assert statements! [which is what I meant by my request to review how "autofails" are done in Lucie's HW3]
• synching versions and perhaps some small edits here and there
• Q11 answer missing an answer in the tester file? failed to update, it looks like

...

Oo... the tester file has not been updated to reflect the new indexing/enumerating of the student facing file... hmm...

Going to pause for a moment here and consider options. [Things are synched up to Q10/Q11 between STA130_HW_7.ipynb and STA130_HW_7_student.ipynb]

I wanted to ask if you were looking at the correct version of the homework because the student file does not have any answers in it and Q11 does have an answer in the tester file. Also, I did do the assert False, hint thing as in Lucie's file.

[pointOfive]

Did not git pull locally after switching over to this branch: was not using the most up to date file(s). Will compare my made changes to the current updates: perhaps all is completed after all (which would be great!)

[pointOfive]

Yes -- exactly right @mistryrohan -- I was not using your up to date versions(!!!), but I have them now as follows:

• STA130_HW_7_student.ipynb is the student file
• STA130_HW_7.ipynb is the tester file Is the STA130_HW_7_tester.ipynb file on MarkUs anything? Or just a rename of STA130_HW_7.ipynb above?

[mistryrohan]

Fairly sure that was just a rename because I wanted to make it more explicit but MarkUs has multiple files with similar names on there right now.

[pointOfive]

Ah the world makes sense again :) very happy to see I was just looking at the wrong (old version) files. Excellent (and a good error to make so I'll be more carefully about this in the future! Always git pull to stay up with the times!).

• Q0: broadway.isna().any(); broadway.drop('top_ticket_price', axis=1, inplace=True) # Can only do this once still shouldn't be run in the tester file. It will either error or change the variable to the right answer even if the student didn't do it! Could change variable names if you wanted to keep a "tester" version of objects around.
• Q1/Q2 (and maybe future ones) are getting automated printouts :)
• Q3: hid test expanded hint
• Q4: slight update/expansion for auto fail tests output formatting
• Small edits to solutions here or there
• Q8/Q9 answer hidden hint slightly updated
• comment out Q9 = broadway[['weekly_gross_overall_in_100k','log_avg_ticket_price']].corr().iloc[0,1].round(3) or answers will not be checked due to overwriting the student answer variable!
• Q10 hid answer; and, changed (hopefully improved) the right answer to the question
• Q11 some finicking with the options here (just standard revisions that happen as a re-read questions
• Q13/14/16 some formatting+expansion for hints
• Q18 answer edit.

This looks great. I'll confirm on MarkUs tomorrow; then, finalize the TUT; then, merge the PR and we're done!

[pointOfive]

Got everything working on MarkUs. Had some problems forgetting to put # test_Qx in tester cells which caused errors and subsequent tester cells to fail. Also had to move # test_Q18 cells up as the last cells of the tester file are not included in the merge after the last matching cell of the student file... Took a while to track this all down an figure out what was happening...