[WIP] How to gather feedback from students

[nuke-web3]

Trial of discussions for UCB happened here: https://www.notion.so/paritytechnologies/Discussions-Questions-and-Feedback-16544e728c474b1c81f38edbaca2b2d5

• public to class discussions
• 100% anon open comments form (no one sees but faculty)

@jonasW3F :

https://docs.google.com/forms/d/1uL5ShkDHf5RORUYy7LY-ncXD7TmlkHfBxKAbhYnNM1s/edit This seems to be a good form to adjust towards our needs

We can use https://cryptpad.fr/ for a truly anonymous form as this seems to be important.

Discussion: https://matrix.to/#/!XpMTBzEdmEEdtZdLkX:parity.io/$VPLhaAMUYIL7O9ay4-3_5SEskpFDBK8kSx4xj1qoU7M?via=parity.io&via=matrix.org&via=web3.foundation

Have ideas or feedback on this feedback? :grinning: please comment here. cc @naterarmstrong @Overkillus

[Overkillus]

Make sure we gauge:

• fun
• difficulty
• usefulness

And some extra optional question for suggestions or unique criticism.

Preferable have one standardised form and use it for each module.

Once developed it needs to be linked in Notion and additionally should be mentioned again at the end of each module. If we want students to complete it it needs to reach them through multiple channels.

[naterarmstrong]

I think 1 per module is probably best. For questions, I think something like the below, although I think it could use 1-2 more questions on the web3 applicability, and maybe could be pared down a little bit. This is heavily based on Jonas's linked form:

Using the parentheses to represent options for the following questions

Module name:

Level of effort (Almost none, little, some, a lot, a ton)

• How much effort did you put into the module?
• How difficult was the module to understand?

Learning (poor, fair, satisfactory, very good, excellent)

• How would you rate your level of knowledge at the start of the module?
• How would you rate your level of knowledge at the end of the module?
• How much knowledge do you think is necessary to pass the course?

Instructor Skill (Strongly disagree, disagree, neutral, agree, strongly agree)

• Instructor was an effective lecturer / demonstrator
• Presentations were clear and organized
• Instructor made the material interesting
• Time was used effectively during class periods
• Instructor was available and helpful
• Instructor answered questions well

Course content (Strongly disagree, disagree, neutral, agree, strongly agree)

• Learning objectives were clear
• Module content was well organized and planned
• Workload was appropriate
• All students were able to participate fully

Web3 Applicability (Strongly disagree, disagree, neutral, agree, strongly agree)

• Module will be useful in a career in web3
• After finishing the module, I feel confident to tackle this topic in a career

What aspects of the module were most useful or valuable? (OPTIONAL)

How would you improve this module? Were there any pain points? (OPTIONAL)

Are there any topics related to this module you wish had been covered? (OPTIONAL)

[nuke-web3]

Think this is perhaps too many granular questions? 15 minutes might be too short if you actually respond to all of them.

[jonasW3F]

I think people are used to those forms and the questions are rather standard. It should be fine for 15min

[naterarmstrong]

We can definitely cut some of the questions from this, but also all of these except the last 3 are in a multiple-choice format, like this. The things in parentheses are the options for the question. So I don't think it'll take too long.

[naterarmstrong]

If we're going to cut some of the questions, I think the ones that are least important are:

• How much knowledge do you think is necessary to pass the course?
• Presentations were clear and organized
• All students were able to participate fully

[Overkillus]

Instructor Skill

How do we tackle modules with multiple instructors? Econ & GT has Jonas and Alfonso so this form will be a bit confusing

[jonasW3F]

As I said, I don't think that it takes students to fill out that form too long. I'd disagree that "Presentations were clear and organized" is not important and can be cut. The other two you wrote are fine to cut.

That brings another thing to my mind: We should make clear that the feedback is on the lectures and not the assignments

[jonasW3F]

Instructor Skill

How do we tackle modules with multiple instructors? Econ & GT has Jonas and Alfonso so this form will be a bit confusing

Good question. Either we make clear that it's an aggregated assessment or we duplicate the questions per instructor. In other modules, how many instructors are there generally? This is feasible for maybe 2 instructors, but if some modules have more than that, the approach might not be reasonable.

[Overkillus]

Aggregate assessment might be quite misleading if one lecturer is really good and other mediocre. Usually 2-3 lecturers deliver 90% of the content so we can safely skip the rest.

For instance I have only 2-3 hours of content in the Blockchain module so don't think I'd expect to have a dedicated feedback for me.

[naterarmstrong]

I support duplicating the instructors section for up to 2-3 instructors per module.

[naterarmstrong]

One other thing that might be useful on this form that isn't in the draft is something for feedback about the specific activities that are run or the overall opportunity for hands-on stuff.

[nuke-web3]

going to try https://docs.cryptpad.org/en/user_guide/apps/form.html for this note that it's a bit fiddly to use but provides strong anonymity, is FOSS, can be embedded in notion and a cool tool to introduce students to.

Known issues we hope get resovled:

• [ ] https://github.com/cryptpad/cryptpad/issues/868

[naterarmstrong]

Some items from the element thread on this:

• Possibly prefer google forms over cryptpad if de-sybiling is an issue
• Jonas: IMO, simple google forms that says that details are not transmitted offer a better mix on privacy/sybil protection but lets see how it works out
• Maciej ran into issues loading cryptpad
• Gauging difficulty of assignment: Uses 2 questions instead of 1 in cryptpad to allow students to show wider range of opinions.
• Feedback on grading (Maciej): 1 question suggestion~ I'd reaaally like to have feedback on the grading as well. IF this would be published after the the grades are out students could also say if they expected the grade they received etc. Then we can see if people received the grades they were aiming for or maybe if there were some pain points around that.
• Cooperation (Maciej): Also since it's fully anon can we add a question about how much they cooperated with others? Would be nice to see the patterns between students that worked fully solo and the ones that were more social. This can help us take better approach in the future when deciding which mode is better. Maybe the reason why the initial modules have lower grades is people are much more hesitant to cooperate than later ones etc

[nuke-web3]

Some settings and communications to students clarifications:

We make it clearly communicated in the form's first text (under the title) that students may identify by their github usernames if they want to - but this is explicitly not required.

~~We configure and communicate so only in a single session do we allow are tweaks on a single submission, so after they get grades they wont come back and have a biased view of the content based on their (poor) grade. We insist only one submission per week and further feedback should not use the same form - they should rather use the anon feedback open ended text form and/or the public class discussions and/or DM the instructors. ... we don't want to encourage hard-to-assess multiple anonn submissions and sybil abuse. ~~ actually we do want to use multiple edits but we want to close the form to responses before grades are issued. If more feedback is needed after this, it's almost always a contesting student on score and needs to be addressed by graders in DM - not an anonn feedback form.

[nuke-web3]

TIL you can see how many people are viewing the form. If we want to do this synchronously with people, we could confine the form to be open only for that time and monitor for when they are all done... :thinking: nice to see activity real time too.

[naterarmstrong]

Form auditor link (THIS MUST STAY PRIVATE): https://cryptpad.fr/form/#/1/view/A2QKuT-ETSA9RhbSG41cKg/vePUgk519NOsTRs7ZusMqnMEKKzwY5dlXU9arJxMIrQ=/auditor=a3k5ZTNzaENJVVdXNXRkWEVTUFhjTS9mYVNlYVRmMmc=/