Collect survey feedback on the year's rule book

[nickswalker]

Is your idea/suggestion related to a problem? Please describe.

Some of the emerging discussions on which tasks to keep for 2020 (mostly off of Github, but #173 is related) have touched on how teams made decisions about which tasks to participate in. While we have some good direct observations from folks active here, and we have things like scores and pre declared participation, we don't have good overall insight into how teams thought about choosing tasks this year.

In general, discussions here will always have an incomplete view of how teams felt about the rules for the year. Another instance where assessing this has come up was on the imperative of reducing the burden on teams (#476). Did last year's changes accomplish this? We don't have a good way to say.

Describe the solution you'd like

An official (optional!) survey emailed out to participating teams (team leader or team representative) after each year's competition gathering their qualitative assessment of how the rules and procedures performed in practice. Can include some general items that would remain the same every year, as well as items targeting areas of specific concern for the next year.

Describe alternatives you've considered

There could be benefits to having a more formal reflection on the rules happen during the competition, but overall it seems like teams are busy and distracted while on site anyways. The closest thing currently seems to be new TC meeting which occurs each year after the finals, but this discussion is mostly about reflections on the logistics of administering the year's tests, and only the subset of teams represented in the committees are represented.

[kyordhel]

Prove me wrong, but I'm not gonna lie. This sounds like an awful idea.

Opinions are easy to be spoken, and not necessarily easy to implement. When it comes to the rules, I disregard any opinion as useless unless is backed by a Pull Request. The rulebook is a public repo, so if someone is really interested in changing anything, can do the changes themself.

Why is this a bad idea?

• Challenges don't look the same to rookies and to veterans.
• Some people are way too noisy, yet they don't speak for their teams.
• [Bigger] Teams with more speaking-persons will be heard more than smaller teams
• Opinions and interest can change from one year to another, leading nowhere. In contrast, following carefully designed roadmaps will eventually lead to achieve the expected goal.
• All teams were invited to join the TC. If teams are not represented there is because their own lack of interest

I'm more for a TC level discussion and the design of roadmaps with scientific value as suggested by @justinhart. Then we use those roadmaps to direct the competition. When carefully designed, they can make @Home attractive to industry and as a mean to test scientific research and benchmark.

Yet again, prove me wrong.

[nickswalker]

1. The survey is not anonymous. TC sees responses in context of the team
2. 1 survey per team, sent to the team leader
3. Same as 2
4. That doesn't change. The proposal is not to change the rule-making process, it is to collect additional data about qualitative impressions
5. There are many reasons why people may not be able to join the TC which are not "lack of interest." Dismissing their input because of these factors does not create a welcoming community. But again, proposal is orthogonal to TC membership

[benatuts]

@nickswalker I agree with your comment about creating a "welcoming community".

To build on what you're saying and respond to the other comment...

As technologists, we tend to focus too much on technology and forget about people. However, if we really want to make big leaps forward in robotics, we need to acknowledge the fact that fostering a vibrant, welcoming and growing community is more important than any particular technical detail of a competition or any line of code.

Assuming that the reason a participant hasn't joined the TC or written a pull request is because of a lack of interest creates an insular community. The assumption ignores many other possible explanations:

• They didn't feel confident that they could make a contribution
• They are starting a new team and weren't at the previous competition
• They were worried about their ideas being rejected
• They didn't feel that they'd be welcomed
• They are worried about graduating on time and couldn't make the time commitment
• They are not confident in their English language skills
• Their team leader didn't mention it to them
• They're not ready to commit to a full year in a TC because they're not sure about their future plans
• They feel they aren't "technical" enough or expert enough about robotics
• They thought they were too junior or new to the competition
• They were afraid of making a technical mistake publicly
• They want to avoid situations with high levels of conflict
• They've had bad experiences in other committees
• They were worried that they'd invest lots of energy only to have it ignored or discarded
• and so on...

Perhaps in a world without emotion, we could say "PoC || GTFO" and "pull request || GTFO" but to build a community, it is important to reach out to people and include them in the process, to be positive and encouraging and to let people know "you belong here regardless of who you are, your background or how much experience you have in robotics". To say "we only accept contributions in these two particular ways" may be much more efficient but sometimes thoughtful inefficiency is what builds communities.

I, personally, would be delighted to receive a survey -- even one that goes to individual participants, not just team leaders -- and to see the perspectives of every participant be used to shape the rule-book (especially the perspectives of new teams!). To me, I would see it as fostering a positive, welcoming, outward-reaching, whole-of-league culture.

[johaq]

To me a survey seems a little too formal. I think we need to offer a space to discuss the rules at the competitions. Some of this is done in the team leader meetings. Problem is, there is always very little time. Maybe we could at least collect email addresses of all team leaders and then ask for free form feedback on the rules.

[nickswalker]

Yes, I agree that it shouldn't be overly formal. If it is, fewer people will fill it out.

Throughout the competition this year, we should keep in mind the kinds of prompts for feedback that we'd want to put forward.