Develop initial set of formative assessments for priority lessons (4-5 assessments)

[schanzer]

Histograms

Q1: Distinguish between bar charts and histograms Q2: Create histograms Q3 + Q4: Interpret histograms Q5: Choose the correct bin size

Box Plots

Q1: Compute a 5-number summary from a dataset Q2: Interpret a 5-number summary Q3: Interpret a box plot Q4: Describe box plot shape Q5: Compare and interpret box plots [maybe we can ditch this question?] Q6: Connect box plots and histograms

• [x] Linear Regression

Q1: Draw and interpret predictor functions Q2: Use and interpret linear regression equation Q3: Interpret linear regression

• [x] Threats to Validity

Q1: Identify selection bias and bias in study design Q2: Identify poor choice of summary data Q3: Identify confounding variables

• [ ] Stretch: composing table operations

[retabak]

Starting a potentially BIG conversation, which might need to happen in real time:

I think these formative assessments should highlight the most common student misconceptions. In writing the first formative assessment (histograms), I consulted the literature on misconceptions about histograms, and I discovered that our stuff doesn't quite address these misconceptions. (See #2085.)

I can ignore the literature and write the formative assessments based on what's covered in our lessons, or.... I can propose revisions to our lessons as I look at the literature. Obviously, I think the latter is the better solution, but it would (of course) be more time-intensive... and might motivate us to do a literature review on all of the various data science topics that we cover. Curious to hear your opinionis re: the feasibility / urgency of this work, @schanzer and @flannery-denny. (I think that focusing on the literature would likely bring us in greater alignment with the Common Core standards, also.)

FWIW: it was VERY easy to find the most common misconceptions about histograms. I just put "histogram misconceptions" in google scholar and read the first four articles. I don't think we need to do any super deep dives.

[schanzer]

@retabak We don't really have a choice this year. The fall2024 lessons are finalized, and cannot be changed.

We need to do option 1, and add notes to https://github.com/bootstrapworld/curriculum/issues/155 so that we can improve the histogram lesson for fall2025

[retabak]

That makes sense. I got pretty excited, but it's true, this would totally be a big big project, not to mention that lessons are finalized.

[retabak]

questions I'm pondering:

• Do we require that all questions are auto-checking? (e.g., desmos does the grading for us!) I am suspecting yes.
• Are we assessing students on their mastery of the coding / Pyret skills? or just on DS concepts?

[schanzer]

@retabak Yes- auto-checking. Just DS concepts

[retabak]

Okay! Here is a draft of the histograms formative assessment: https://teacher.desmos.com/activitybuilder/custom/6671db339409cfb8815bf875

There are a variety of considerations / questions / concerns that I want to share:

• I did my best to align the assessment with the content of the lesson. The four workbook pages in the lesson are on these topics: Summarizing Columns with Bar Charts & Histograms; Making Histograms; Reading Histograms; and Choosing the Right Bin Size. You'll see that the titles of the desmos slides correspond with these worksheet titles. I opted to include 2 questions on Reading Histograms; the first question is lower level, the 2nd question (multiple select) is more difficult.
• Given that we teach students to make a histogram, I thought they should make a histogram (Q2). That said, I don't know how to have that portion of the question be self-checking... but the table is. We could just have them do the table, or we could do a MC question where they choose the histogram that matches?
• I am NOT a pro at using the Computational Layer, and it is very time consuming for me to write the code to make the questions score themselves. Maybe I should invest some time to actually learn how? (I'm a hacker at the moment, just copying other people's code.) Or, alternatively, would it be super duper simple for someone else (Dorai / Emmanuel?) to invest some time in learning desmos CL? (Truly, I have no idea.) I didn't (yet) figure out how to throw error messages to students, such as, "You've entered a word - please just provide a number" -- but I suspect that it is possible.
• On that note, I only know how to get Desmos to indicate if ALL elements of a table are correct, so that is what I have done. If we feel strongly about awarding partial credit.... welp, I'm not sure how we will proceed. The student's screen can indicate a check or an X for each response, but the teacher dashboard cannot.

How are we actually going to access this data? Will we be able to see teachers' dashboards? The teacher dashboard is actually not AT ALL easy to use for data analysis. It doesn't do... well, anything, except give a checkmark or an X for each student's response. In designing the assessment, I think we do need to consider how we will be receiving this data.

[schanzer]

@retabak this looks AMAZING!! Yes yes yes! I think choosing the page titles was a good idea. I wish I knew some CL, but that's not going to happen any time soon. :(

As for how we're getting the data: Dan says we can send them an activity URL and a list of teacher Desmos names, and they'll send us back all the student data associated with that activity.

I think this is a really strong start. Let's replicate this for the other lessons this week!

[retabak]

re: Dan says we can send them an activity URL and a list of teacher Desmos names, and they'll send us back all the student data associated with that activity.

Cool! I'm wondering if there is there a way to find out if teachers have made a copy + edited the formative assessment activity? Many regular desmos users like to customize things - e.g., they add an intro / check-in slide, they put the assessment at the end of some other sequence of activities, etc, etc. Wondering if there's anyway we can still gather this data - or if there is a way to encourage teachers to use OUR activity, without modification?

[flannery-denny]

@retabak Great start :)

• [ ] I think we could improve the wording of slide 4. For me, most means something different from the majority so I would find myself guessing about whether or not slide 4 is a trick question.

• [ ] I also don't see the answer that I would want for slide 5, which would be something less than 100, but not necessarily between 20 and 50, since I can't play with the histogram to know whether or not 50 would be too small.

• [x] From the checklist at the top of the issue, it looks like you've drafted a second assessment, but I'm not seeing a link. Please add links next to the items in the checklist as you check them off so that it's easy for anyone reviewing this issue to find the most updated version of things. Thanks!

[retabak]

@flannery-denny -

I also don't see the answer that I would want for slide 5, which would be something less than 100, but not necessarily between 20 and 50, since I can't play with the histogram to know whether or not 50 would be too small.

I hoped students would notice that the largest weight is ~200 or less... which means that a bin size of 20 would produce ~10 bins, and a bin size of 50 would produce ~4 bins. If you were unable to notice / reason through that, then it obviously is too hard and needs to change. Perhaps I should invite students to play with the histogram. (There is a hyperlink to the pyret starter file - but not an explicit invitation.) Would that be adequate? That said, @schanzer said we don't want to test coding ability, so perhaps the premise of this question is faulty to begin with.

I think we could improve the wording of slide 4. For me, most means something different from the majority so I would find myself guessing about whether or not slide 4 is a trick question.

Lol. For me, "most" and "majority" mean literally exactly the same thing, but I have no objection to you playing with the wording.

And: I've added a link to the histograms formative assessment for your review. (See top comment in this issue!) I have not added in CL stuff, but I'm realizing I should probably save that for the end.

[flannery-denny]

@retabak Here's feedback on boxplots

I'm not able to edit from teacher success and am going to wait until I can to confirm that we got desmos set up correctly.

• [ ] I'd suggest we put the directions re answering with NUMBERS into the column header for both box plots and histograms

  

• [x] If my understanding of box plots is correct, on slide 2 of boxplots we should change "more than" to "at least"

• [ ] slide 3 -- there are two symmetric box plots and it's not possible to match them both to the word -- suggest changing "the four sections of the box plot are equal in size" to something about the data itself, like "the data appears to be evenly distributed across the range" -- similarly, suggest changing "there is a long whisker trailing off to the left suggesting potential outliers" to something less leading, like "there are potential outliers at the low end of the range"

• [x] for slide 5 I would get rid of the words about outliers, as they might lead a student who understands box plots well, but doesn't know how they translate to histograms.

For me, majority is a mathematical term with a clear definition of "more than 50%" and most is a conversational term. When I say, "most of the time Dacks is asleep by 9:30" I mean that it is an exception when he is not. But @schanzer may disagree.

[flannery-denny]

@retabak Seems you've been pretty interested in this issue, but I know you have a lot of AI on your plate. Checking in about whether you imagine drafting the rest of these this month or are hoping not to have to? Unless @schanzer has rethought this, we need to wrap these up in the next two weeks.

[retabak]

I do think we should have a conversation as a team about the assessment drafts that exist so far before we create more of these. That said, I would love it if you wanted to work on drafting the remaining formative assessments. (I'm not sure how much you have going on at the moment, though!)

On Fri, Jul 12, 2024 at 2:57 PM flannery-denny @.***> wrote:

@retabak https://github.com/retabak Seems you've been pretty interested in this issue, but I know you have a lot of AI on your plate. Checking in about whether you imagine drafting the rest of these this month or are hoping not to have to? Unless @schanzer https://github.com/schanzer has rethought this, we need to wrap these up in the next two weeks.

— Reply to this email directly, view it on GitHub https://github.com/bootstrapworld/curriculum/issues/1789#issuecomment-2226188967, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUY3SE5MH6CENH6SGTKPKS3ZMARIPAVCNFSM6AAAAABCJG5TICVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRWGE4DQOJWG4 . You are receiving this because you were mentioned.Message ID: @.***>

[schanzer]

I'm sprinting flat-our through the 22nd. I just wrapped up PD in Bay City and I'm at the airport frantically updating stuff for Algebra 2 next week.

I'd hoped to have drafts ready early enough to give feedback before PD started, but we are where we are and I think it's best to draft the rest so we have something more solid to work from when the dust settles.

[flannery-denny]

To be clear, if we were not planning to collect and distribute aggregated data from these assessments and desmos were capable of sharing more nuanced information about student responses, I would love what @retabak has drafted. But given her astute observation that the information we can see on the teacher dashboard is fairly useless, combined with our intention to use aggregate data at scale (and give district administrators the capacity to see aggregate data at scale), I believe we need to choose other formats for these questions that will give us reliable and useful aggregate data. I expect the aggregate information we would get back from desmos via the current formatting would be misleading, which is far worse than useless. I encourage the team to think about our conversation with Paul Zachos about what useful educational information looks like. The potential for generating bad data that will be misused has been my concern about committing to building out assessments for each of our lessons from the very beginning of this discussion a year ago.

[flannery-denny]

I've been thinking more about the checkbox question. Makes most sense to me to convert it to multiple true or false slides. We don't lose anything, since the original question was also just asking students to consider each statement individually and decide whether or not it's true or false, and this way the data collected would become precise and valuable.

[schanzer]

I'm inclined to agree, @flannery-denny. While seeing 20 slides might seem intimidating at first, as soon as students realize that each slide is a single question I imagine they'll calm down.

[retabak]

Here's the new and improved 20-slide long box plots formative assessment! Also added a question about computing the five-number summary.

https://teacher.desmos.com/activitybuilder/custom/668d7be8fd7c6212638763d6

Feel free to offer feedback or change it yourself, @flannery-denny and @schanzer. More coming...

[retabak]

Here's a linear regression draft:

https://teacher.desmos.com/activitybuilder/custom/66bf6a408a890dd0d0ea2654

It is only three questions at the moment. Based on my overview of the lesson, I think they are addressing the right skills. We said we'd aim for 4-5 questions, so I'm wondering if I should add another question on one of these 3 skills, but using a different dataset.

Curious which question you think is highest priority, here? I'm thinking Q3, esp since there is a single slide on that skill at the moment. It is very annoying finding the right datasets, though!

[retabak]

Here is a "threats to validity" draft: https://teacher.desmos.com/activitybuilder/custom/66c391955b3af1281567a903

Noting that it was tricky to write this one, given that every single worksheet we have in this lesson is exclusively open response. We can't do open response because desmos can't score open response.

I thought it might be appropriate to have students choose the threat to validity that is described in such-and-such scenario, but quickly threw out that idea upon remembering how much overlap there is between the different threats (ie, when there's selection bias, there is also generally bias in the study design).

The sort of question I did write (which of the following represents such-and-such threat: option A, B, C, or D?) is NOT practiced in the lesson, so it feels like I've written a not-entirely-fair assessment. That said, I couldn't think of a better solution. Maybe one of you can!? Or perhaps this is fine.

[retabak]

@flannery-denny - just pinging you to let you know I've updated the topmost comment in this issue, as requested