Created HW9

[Bortoise]

I created the homework for week 11. I also updated the Readme with a link to the new HW.

[Bortoise]

There should be an image rendered with the last question, it is in the folder. It should work when downloaded locally. Alternatively I could have it hosted somewhere and just link it to the url, but I prefer this way.

[lucieyang1]

Hi @Bortoise, can you upload/send the tester file for the homework? Thanks!!

[Bortoise]

Hi @Bortoise, can you upload/send the tester file for the homework? Thanks!!

I'll send it in slack dms, I'm also planning on setting up the markus autotester later today if you want to test that out.

[lucieyang1]

Ok, I just finished the homework. It took me slightly longer than 2 hours, but I was also trying to learn graphviz and fixing a bunch of issues I ran into. It'd probably be a good idea to add some hints in the questions (like putting the clf.predict() hint in the questions instead of in the tests)

I was a bit confused with the train-test split. Q1 asks to divide the data into train/test, and gives the template train, test = None. Does that actually work? From what I can see, I think X_train, X_test, y_train, y_test should be used instead. Also, we might want to move the train/test dividing into the second question, because I think you need to know the X and y parameters before dividing?

Q4 asks to use the same train/test split as previously, but I'm not too sure that's possible, given that we are using different X parameters and would need to drop different na values. So for Q5, I got different values for both models.

[Bortoise]

The problem I ran into when setting it up the X_train, X_test, y_train, y_test way is that I don't know how you would drop the right rows with .dropna() when dealing with several dataframes at once. On the other hand this is simple to do using test[["life_exp_good", "social_support"]].dropna() in the way I set it up (this is also how Scott does it in the corresponding lecture). Let me know if you know a way around this.

The reason we set up the train-test split before Q2 is that we want to use the same split even when we are making more classifiers with different inputs, so we want to compare apples-to-apples. You are supposed to drop different na values for Q5, hence the setup. I will try to add additional explanation to make this more clear.

[lucieyang1]

Here's the code I had:

selectfeatures = happiness2017[["social_support", "life_exp_good"]].dropna()
X = selectfeatures[["social_support"]]
y = selectfeatures[["life_exp_good"]]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

It seems like you made the split before dropping na values. I dropped the na values first, which is what I think was done in the video. I was thinking, if you drop the na values after making the split, then the split might not reflect a 80/20 split of the data anymore.

[Bortoise]

I see your point about the issue of the 80/20 split. In this case I don't think there is much of a problem since the split being exactly 80/20 isn't really important at all, but it's not best practices in case there were many na entries. However, I really do want them to use the same train-test split for both trees. To remedy this, I'll have them drop the na values for all the relevant cols before doing the train-test split. Thanks!

[pointOfive]

Going to go ahead and add my revisions/updates/commits to the HW and tester files now. This will lose some of the updates made while I was revising files; but, I am testing things on MarkUs now and I think I am aware of what changes might be necessary to get things working correctly on MarkUs.

• note about deleting and adding cells has been edited (and it's good one that we might want to eventually add to other notebooks)
• Improved marking instructions
• Created Q1 problem and moved old Q1 to Q0... thought about this last time around, but decided it should def be done this time around so pulled the trigger on doing it this time: generally I we'd like to test things at each stage to help the students pinpoint where they're going wrong whenever possible
• fixed Q3 hint which was accidentally wrongly copied from another question
• tester Q4 added formatting for automatic test failures (you did add these edits yourself after I started editing, but they were not in the tester file I had access to downloaded from MarkUs)
• Q6/Q7/Q8 I think Lucie was correct: a cell for students to start working their answers in will be helpful for them methinks
• Been making some other small changes here and there where it strikes my fancy/sensibilities to do so... these can be seen by pursuing my commit below
• Lucie's comment that decision tree labels (both earlier and later in the notebook) should be ["Bad","Good"] is correct (corresponding to the more natural ordering of [False,True]). Please do take advantage of helpful comments like this in the future... it seems Lucie's comments have been ignored and disregarded
  • Added a comment to Q6 help clarify this for students as well
• Q7 needed to have students remake the decision tree visualization if they're going to repeat Q6 which also asked students to confirm the predictions from the decisions trees
• I decided Q8 should provide some additional help hints for students: try to put yourself in the students shoes in the event that they know nothing (and haven't seen my lectures, for example)
  • Also added Q8 = (clf2_sensitivity, clf2_specificity, clf2_accuracy) + (clf3_sensitivity, clf3_specificity, clf3_accuracy): I do want this full formality finalization/polish for consistency for students
  • Same story for Q11 and Q13
• Expanded feature importance section a bit to my liking
  • And the subsequent Q9 was a little barebones so I expanded that a bit
  • Needed to add plotly as a dependency for MarkUs as I added a plotly figure to the Feature Importance problem
• Q10 good question and hint -- expanded only ever so slightly
• False positive/negative rate definitions were still mixed up (so I fixed that); but, answers to Q11 were still right
• Q12 is a good question: I don't want to expose the answer and I made few preferential edits

When reviewing your changes made while I was editing I did not always change to the "hidden" style of your MarkUs output, e.g., some early questions, Q8, etc.: sometimes I decided it was good to expose a little more of the answer (but, I did see those changes you made and generally approved)

There were concerns about setting random seed: at the moment I have not changed my files to reflect this and will be running things on MarkUs to see if I encounter any issues, so I have not made the changes corresponding to our conversation at the following points:

• random_state=1985
• ".70,.71,.72 instead of .49,.50,.51"

I did add some additional output into the tester file to help with looking into this in the form of print("Answer Submitted:", Q13)

[pointOfive]

print("Answer Submitted:", Q13) did not work -- fixing

[pointOfive]

Running into a 5mb Markus file size constraint due to the graphviz images -- accommodating

[pointOfive]

Once I make these changes in the HW file(s) commit that will eventually appear below, to see my initial changes relative to the your last version of the file (available on GitHub) you'll need to go click on the specific commit and see the changes highlighted in the file. It seems to me this is a fairly good way to get a good view of what's being changed how even thought it's "GitHub texty"

[pointOfive]

Running into this error again: "Cannot display this file. Please download it to view. ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration."

[pointOfive]

The error is caused by graphviz decision tree (XML, I think) figures. The nbconverter to html for rendering in MarkUs chokes with the above warning. Spent a couple hours looking for potential workaround; and, sadly, best option at the moment seems to just be to make clear instructions to students that their final notebooks must not have any figures showing. Very hacky; but, it is something that I can change right now and will be a sufficient placeholder.

[pointOfive]

Correction/Update: exciting news -- MarkUs folks got back to me and nbconverter will not fail in an updated version. Currently it appears it does not render the figures, but this is not important for TA workflow. Very pleased to have sorted this out with relatively minimal time loss. So this will fix itself whenever the next update of MarkUs is released (which will be in time for the Fall semester)

[pointOfive]

I didn't have any issues with the concerns of:

• random_state=1985
• ".70,.71,.72 instead of .49,.50,.51" so I'm not making these changes at this point.

[pointOfive]

Looking for one more confirmatory review from both Matthew and Cole.

[Bortoise]

Ok I did a quick review and I only noticed one major issue. In Q1, the setup was a bit confusing so I moved the explanation about subsetting before dropping NAN values above the line about motivating the choice of 1985 as the random seed.

[MatthewYu06]

Homework looked good and made sense, took me a little over two hours (but I was also noting edits as I went through)

No major changes

• added some question clarification to Q6 and Q7
• clickable link to matrix cell in Q8
• small typo fixes that I caught throughout

[MatthewYu06]

also looked at tutorial, looks good so no edits made

[pointOfive]

TUT was very nice. Was happily and pleasantly surprised at how engaging and fun it was. I think the students will likely find this to be very compelling due to the high quality of the examples.

• I have nonetheless made quite a few edits around personal preferences
• Relevantly, I have changed the "down/next" slide/subslide layouts and usage...
  • You may not agree with my slide/subslide/etc. changes; but, I think it flows better this way; because, I think there's a risk of "down" slides being accidentally missed by the TAs (since there's no down button but you have to click it and it's pretty easy to just click right ); so, the changes (including more explicit instructions on how to navigate the slides) are meant to make the slide flow a little better... I would not be opposed to you reviewing and making additional changes that you think could further improve this.
• Transformed the initial part of the group discussion into "Tutorial Activities" (and created some answers to review)
• Re-sequenced this material to earlier in the deck as I (a) don't think the "rights" part is necessary to address the initial considerations and (b) I think they add too much "density/complexity" to these activities, and (c) I think this makes for a transition into these topics that naturally "grows and expands" into the ethical considerations/topics.
• We also have to be careful, because we're not addressing "thresholding" to increase/decrease/control the number of positive predictions, which is how we would really approach looking into the FP/FN tradeoff: you can't really "prioritize" sensitivity or specificity when you just have those numbers in front of you -- I mean, they are what they are, and all you can only just talk about what they mean... they are only "prioritized" in the sense that you would be more or less concerned about one of the other.
  • I have made changes to address my concern here; so, if it seems like I'm just splitting hairs, the specific nature of my concern will have been addressed through my edits
• I have merged the final exercises (skipping the "group" activity aspect of Case Study 2) to make the "Tutorial Assignment"
  • I am concerned about time as follows: 25min through case study 1; 20min for small group activity; 30min ethics primer through impact+rights; 15min final quiz (finishing Tutorial Activity); all of which takes us to 90 minutes, leaving maybe 10-20 minutes to finish... so I think we need to just use that to set up the "Tutorial Assignment"
  • I do realize this means students will have to consider the "impact+rights" concepts for the Mortgage and Med school applications / features "cold turkey" without any practice on these sorts of considerations (besides the lead up); but, I'm currently feeling like just having the students get introduced to these terms and explanations is enough and somewhat self-explanatory... at any rate, I'm okay with just having them explore these concepts topics without a tremendous amount of guidance at this point...