trust1995
commented
1 year ago The idea of self-assessment of different role members is really good. We need to think about which ideas can be borrowed from agile and which can't possibly work in a DAO. Honestly, it seems that our structure is too unique to be agile-able. We'll need to structure talks carefully and ahead of time for it to be meaningful in terms of both time and coherency. Also, everyone needs to be perfectly clear about what the intentions of the retrospective are, i.e. it is not an extra-time judge QA session.
Ideally, every role-member will bring everyone up to speed on how things looked from their end. This way we have a summary of the entire pipeline which will make it easy to identify the friction points which need to be addressed. The meeting structure could be something like (rough draft):
- Opening words from judge
- Overall attendance and report quality
- Interesting properties of this particular contest/codebase
- Description of top 3 most contested findings. Explain both sides and rationale behind severity. Add if any precedents should be made
- Optional - Sponsor's take
- Most valuable findings
- Any aspect that could be improved (better POCs, QA-report, underreported contracts etc)
- Feedback on interaction with wardens, mid-contest and backstage
- Feedback on interaction with judge
- Lead warden's take (chosen by community voting)
- Feedback on sponsor
- repo preparation level
- responsiveness
- productivity bottlenecks
- Any abnormalities during the contest
- Objective feedback on judging
- responsiveness
- performance in contested submissions
- Secondary wardens' take (max +2 additional wardens)
- Reflections & proposals for improvement (anyone)
Importantly, at least one member (staff or volunteer) will need to summarize the meeting and have it available. This would be crucial to measure progress and pass on any relevant knowledge to others who were not on the call.
GalloDaSballo
C4 operates competitively, but it's members are also operating as a team whose goal is to make a sponsor's project more secure. Wardens, Judges, and CAs all perform roles throughout the contest with these goals in mind. Often there are distinctions gained or suggestions about how to make the process more effective which are surfaced during a contest. In agile software development we have a process, called a retrospective meeting, designed to surface the successes and failures of the team over a certain period of time. I propose we add this concept as the final stage of C4 contests.
How would this work?
Retrospectives are designed to be a blameless self evaluation where self is the team as a whole. Atlassian has designed a playbook for these meetings which is adaptable to the process we have at C4.
The stages of a retrospective meeting are actually quite simple. Team members (judges, wardens, and the contest's CA or CAs) start by writing down individual notes that fall into the category of "what we did well" or "what we can do better", with team members provided 15 minutes for surfacing notes in each category. "We" in this context is the entire team, not individual members. It's generally advisable that each participant come up with at least one idea in each category.
Once these notes have been gathered the team works to group them into themes, similar to our deduping process. These themes are then discussed by the team for 10 minutes. A further 10 minutes is taken to discuss specific actions which can be taken to improve future contests. Traditionally individual team members are assigned to actions, but in our context these will probably inform process changes that are adopted in future contests.
Frequency
I suggest that we start with one or two contests to see how the process translates and am volunteering to do this for contests which I judge. Once we as an organization have had a chance to tweak the process in ways that make sense for C4, I'd recommend that we do this for at least one contest per month. The more controversial the contest, the more effective the process is likely to be.
Participation
I envision participation being open to all judges, all C4 staff, and any wardens who have participated in the contest for which the retrospactive is held. Participation in the contest for wardens would be defined as having submitted at least one issue or having commented during post-judging qa. Optionally, we could invite sponsors to join this effort if they want to.
Scope
The scope of each retrospective would be the contest in question and nothing else. Past performance of anyone involved should not be a subject of the retrospective, mostly because any past performance issues likely involve a different team.
Potential issues with the process
Agile development teams typically consist of no more than 8 members (after including the project manager and product owner). C4 contests sometimes involve more than 100 people. For this reason it may prove necessary to limit the participation or run multiple retrospectives for each contest.
In software it is usually fairly easy to track action items. In C4 doing so may be harder given that not everyone involved in the retrospective will participate in every future contest. It is also likely that many of the actions will fall on C4 staff or people who were not part of the specific retrospective (for instance, all wardens or all judges). This will likely require some fine tuning specific to the C4 structure.
Many times a neutral third party is helpful in retrospective meetings to help keep members on track and suggestions blame free. The nature of C4 may make it hard to find someone who is truely neutral in this way. It's possible that a judge or C4 staff member may be able to take this role, but it may be that we need to add a category of participant to the process who acts solely to run these retrospectives.
Thoughts?