[committee] Potential survey of community members about lessons

[annajiat]

Should we conduct a survey for

• potential new lessons
• popularity/utilization of existing lessons
• how the lessons can focus on being more helpful to the other lessons in sequence in workshops

[bsmith89]

This is an interesting idea!

Would this survey be mostly targeting instructors or learners, in your opinion?

[annajiat]

Thank you @bsmith89 for the encouragement, I believe the primary target would be the instructors, helpers and workshop hosts. We could design it in a way that the learners and others could provide feedback too.

[bsmith89]

Looking forward to discussing this more during our meeting.

Are you (or anyone else) aware of existing data on:

• which SWC lessons instructors teach the most (which of the python or R lessons)?
• how they adjust them (e.g. which chapters are omitted for time, which of the exercises are preferred)?
• or how often the experimental or in-development lessons are taught (e.g. alternative Git lessons)?

[richmccue]

Another thing that might be useful to add to a survey would be a question or two about the prioritization of lesson plan elements as well as possibly some workshop disaggregation to make it easier for instructors to customize the workshop content based on the needs of the learners and available time.

[bsmith89]

Perhaps we can continue to use this thread and some meeting time to compile a list of what data we'd like to collect (and from whom).

If we have a pretty specific request, I'm hoping that we can pass that off to the Carpentries core team to produce the survey and distribute it.

@ErinBecker, @TobyHodges, is this something that there would be bandwidth for?

[tobyhodges]

if anyone in the group is good at webscraping, we may be able to provide a list of workshop website URLs - from which one could pull out the Schedule/Syllabus tables each time...

[bsmith89]

Is this schedule scrape perhaps similar to the data that Erin said was now available?

[tobyhodges]

If I understood correctly (@ErinBecker please speak up if I got this wrong!), Erin was talking about data collected when people request a workshop and/or report a self-organised workshop. The form we provide includes a question about what kind of workshop is being taught (see. e.g. "Which Carpentries workshop are you teaching?" in the form for self-organised workshops).

As Annajiat mentioned, some Instructors are nevertheless "mixing and matching" episodes from e.g. the two Python lessons, and that information would not be captured by these form submissions. The web-scraping approach would probably provide a more accurate and detailed overview of exactly what episodes were (planned to be) taught at each workshop.

To be honest, the web-scraping approach probably adds only a small benefit on top of the form submission data, which I would expect to give you a fairly decent summary on its own of which lessons are being taught. But I mention it just for the small chance that someone on the committee thinks they could perform this kind of scraping without much effort!

[richmccue]

Going forward the new stats package would give us information on which workshop modules are being used most heavily, although probably not which modules are being combined together by instructors on an ad-hoc basis.

[bsmith89]

I could see some value in unstructured answers also where instructors could quickly summarize their adjustments to the lessons. (As opposed to asking e.g. for detailed answers about each topic in each lesson.)

Would definitely be biased in who takes the time to write it out, but I still think it would be interesting to know e.g. that >=20% of instructors skipped a particular shell episode or something.

[bsmith89]

There was consensus during our Q4 meeting that this planned survey is worthwhile. @tobyhodges has also indicated that core staff have been discussing "developing a programme of assessment for Carpentries curriculum". Seems like now is a good time to get involved with that process.

@annajiat , while you missed our Q4 meeting, you've indicated to me that you intend to stay involved in the CAC. Attendees at the meeting also wanted to know to what extent you would like to spearhead this survey development, as it was your idea initially.

Who else would like to take charge of pushing this survey forward? (Either leading or co-leading depending on Annajiat's plans.)

There was also general consensus during our Zoom meeting that this survey should, at its core, be designed to find out what lesson materials are actually being taught, and based on what constraints. Specifically, we want to know what materials are being cut out of workshops, and how this depends on workshop format.

The goal of this survey will be to inform lesson development going forward, as well as standardized teaching practices (i.e. what do we tell new instructors about culling materials for time).

[richmccue]

I'm happy to assist @annajiat with the survey.

[mxiang1]

It might also be interesting to ask instructors to indicate the most popular content (episodes, topics etc. with most questions asked?) within the lessons they teach, so future instructors will have a better idea of what learners find most useful for their research.

[annajiat]

Thank you, @bsmith89, for asking. I would prefer to support a leader instead. If needed, I can try spearheading it as well.

We need to take a note that even after preparing the schedule, there would be some changes which may not be reflected back to the schedule on the workshop website. But that is some inaccuracy that we might be able to live with. At least, that may be more precise compared to the workshop request data.

Coming to think of it, it might be interesting if we can compare these 3 data (workshop request, schedule on site, and instructor survey) to see how the plan evolves. However, that may be a separate project someday.

[annajiat]

We can build on @mxiang1 's idea to provide multiple popular templates for the schedule.

Currently, alpha/beta lesson don't show up in the schedule by design. While it has its good sides, ensuring that the instructors are fully aware that the lesson is still in development. It also leads to some repeated extra efforts for modifying the schedule again and again.

In another perspective, a popular lesson needs to be kept updated to remain relevant to an interdisciplinary audience while serving as an example of dissecting and improving design of remaining lessons.

[annajiat]

As a follow-up related to workshop format, during the last two months, I was fortunate to have been involved with all three formats: in-person, online, and hybrid workshops. We may consider accommodating different workshop formats in our surveys and decisions.

[bsmith89]

We may consider accommodating different workshop formats in our surveys and decisions.

Yes! This came up in our meeting discussion as well. The material that instructors cover is highly context-dependent and in particular workshop format matters.

---

I also want to raise the issue of this survey becoming unmanageably large/complex. My inclination is to try to maximize the actionable information that we can get from as short/simple of a survey as possible. In particular, a shorter survey will help us to get a larger and more representative sample of instructors.

@tobyhodges has mentioned information gathering initiatives by core staff, which might encompass our goals as well. I'm curious what the timeline for that is, and whether we should try to work through that mechanism instead of doing an independent survey.

One option would be for the CAC to design an independent survey in the next 6 months but with a much reduced scope (e.g. what parts of the lessons are instructors cutting for time?) and then also contribute to a broader "programme of assessment for Carpentries curriculum" with the more involved questions.

[klbarnes20]

Hi everyone, in case we haven't met, I'm Kelly and I recently moved from Deputy Director of Instructor Training to Director of Assessment at The Carpentries. I'm very glad to see this conversation happening. I have also been speaking to @tobyhodges and @ErinBecker about how we can begin to understand what lessons (and parts of lessons) are being taught. This is a question that is relevant across our Lesson Programs and curricula, so I'm hoping we can take a coordinated approach to how we decide to answer this.

As a first step, perhaps interested members of the CAC could meet with @tobyhodges and I to discuss what a survey (or other types of data collection) might look like. Let me know if you are interested in participating in such a meeting and I will send out a whenisgood. Thanks!

[mxiang1]

Hi Kelly,

Count me in. I’d be happy to discuss more about the survey.

Best, Menglan

On Jan 5, 2023, at 10:31 AM, Kelly Barnes @.***> wrote:

Hi everyone, in case we haven't met, I'm Kelly and I recently moved from Deputy Director of Instructor Training to Director of Assessment at The Carpentries. I'm very glad to see this conversation happening. I have also been speaking to @tobyhodges https://github.com/tobyhodges and @ErinBecker https://github.com/ErinBecker about how we can begin to understand what lessons (and parts of lessons) are being taught. This is a question that is relevant across our Lesson Programs and curricula, so I'm hoping we can take a coordinated approach to how we decide to answer this.

As a first step, perhaps interested members of the CAC could meet with @tobyhodges https://github.com/tobyhodges and I to discuss what a survey (or other types of data collection) might look like. Let me know if you are interested in participating in such a meeting and I will send out a whenisgood. Thanks!

— Reply to this email directly, view it on GitHub https://github.com/swcarpentry/curriculum-advisors/issues/2#issuecomment-1372584980, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJZY4I74QX4IPZHQ6AYSNH3WQ4HRTANCNFSM5QLLIEBQ. You are receiving this because you were mentioned.

[bsmith89]

I'd love to join, too!

[klbarnes20]

Hi all, thanks for your interest in a meeting. I have created this whenisgood poll so we can find a time that works for everyone. Tagging those who said they were interested, but anyone can feel free to join. @tobyhodges @bsmith89 @mxiang1

[klbarnes20]

Wow, you are all fast! Thanks for filling out the poll! I have scheduled the meeting for Monday January 23 at 8 pm UTC (Find the time in your own timezone). If you filled out the poll, I sent you an invite using the email address on file with AMY. If you didn't get it or would like an invite, let me know! Looking forward to chatting with you all!

[klbarnes20]

Hi everyone, thanks for a great meeting today. For those who couldn't attend, I wanted to provide a little summary and a link to the meeting notes. If you are interested, please read the notes and share any feedback you have here. I'm excited to work on this with you all!

We discussed that the objectives of this work will be to:

• Learn what Instructors are teaching/skipping (Episodes, sections of episodes, exercises)
• Understand why they skip the things they skip
• Convert this information into guidance for Instructors on what to teach and what to skip

We talked about the best way to get this information and determined it is to take a few different approaches:

1) a short survey asking instructors what episodes they skip and why with open ended space for people to add any additional comments or information 2) Test the Use of Hypothes.is (a tool that allows groups to collaboratively annotate webpages) with a small group of Instructors to collect more detailed and granular information about the parts of episodes and exercises that people skip 3) Look at analytics (pageviews/time on page) for lessons/episodes to further understand which are more or less popular

Next Steps

• My next step is to draft a survey and share it back with this group for feedback.
• We will also begin thinking about how we could use hypothes.is. I will write up a proposal for how this could work. @richmccue uses Hypothes.is for some of his workshops and said he could share an example of the type of information he shares with his learners.