CA: There is no reputation/merit system for CAs/vCA, and CAs/vCAs ratings are all weighted in the same way.

[stephen-rowan]

CA: There is no reputation/merit system for CAs/vCA, and CAs/vCAs ratings are all weighted in the same way.

Trello Card

User journey (Guidelines )

┆Issue is synchronized with this Trello card by Unito

[stephen-rowan]

CA: There is no reputation/merit system for CAs/vCA, and CAs/vCAs ratings are all weighted in the same way.

[kenricnelson]

During the Fund 7 assessment period, the lack of a CA reputation system became a critical problem. Several assessors used templates to cut and paste repetitive assessments that did not add value to the proposal assessments. This created several problems for the veteran Community Advisor process. First, vCAs began documenting and alerting other vCAs using Telegram Discord. While the intent of this communication was to facilitate vCAs to concentrate on legitimate assessments, two further difficulties arose. First, with hundreds of poor quality assessments, simply adding a "filter out" review became an efficient way to complete a lot of reviews. And second, questions arose as to whether the list of suspect assessments was accurate and whether the propagation of a suspect list was undermining the vCAs working independently to complete their own reviews. These issues have highlighted a critical need to establish a basic entry-level requirement for being a Community Advisor during the Fund 8 round and to continue developing a reputation system that can encourage the development of and the rewarding of high-quality assessments.

[kenricnelson]

Raw input from the community regarding improvements for the CA process: https://docs.google.com/document/d/1ZCgivRkkwEe4VHPLIa1D7nHhwnXB3_9g_vwwQN8NcSo/edit#heading=h.g5p6c42acklc

Spreadsheet dropbox managed by Dan Ribar to consolidate and track ideas for improving the CA process: https://docs.google.com/spreadsheets/d/120HylsMq-2oXeop7m8vBoaf7eb2D8viJazJs_EPWU2I/edit#gid=0

[kenricnelson]

It is critical that Fund 8 focus on improving the quality of the Catalyst process. While 2021 was focused on launching and scaling the Catalyst program, there is now a need to establish quality requirements at each stage of the process. With this in mind, an ATH meeting was held on January 19th to provide input to IOG regarding feasible changes for Fund 8 that will set minimum requirements.

For the proposal process, the group discussed: a) having the Challenge Teams set maximum and minimum budgets per proposal. While there are concerns about the effects of max and min budgets, there is also concern about the effects of processing a high volume of assessments. b) completing a vCA preliminary ranking of proposals and only allowing proposals ranked above 1.5 advances to assessment. Although not discussed, this preliminary requirement threshold could also be completed by the Challenge Team. Perhaps during the drafting stage of the proposals, the Challenge teams could be providing feedback as to whether a proposal meets a minimum threshold. This feedback could be early enough for the proposal team to respond with improvements but if not met by the deadline, then the proposal would not advance to the assessment stage.

For the assessment stage, focus was paid to the first steps toward a reputation system. This discuss was framed in terms of what are minimum requirements to be a CA and vCA; and what limits should be placed on the number of reviews a CA can complete in a fund round prior to becoming a vCA. The results of the discussion are:

a) a training requirement is needed for the CA role. This should include provision of a clear identity. While CA anonymity will be preserved regarding the review process, the registering of an identity for the role is important. Scaling of the training programs, using perhaps a multiple choice questionnaire is important.

b) A poll was taken regarding limiting the number of assessments completed by a CA was taken: Unlimited - 7 votes 10 - 20 - 4 votes 5 - 10 - 0 votes 1 - 5 - 0 votes

Nelson's response: "Although a limit on CA assessments is unpopular, it is necessary to reduce the effect of bot attacks and refocus the CAs efforts toward excellent reviews rather than a high volume of good reviews."

Two requirements could be established to be a veteran CA, a) a minimum number of excellent reviews over the past 4 funds and b) a maximum percentage of filtered out reviews.

a). Since the excellent reviews are still rare (a quote of 6% of assessments was cited), the minimum number would have to be low, perhaps just one. This requirement would have to be over the past several rounds since vCAs do not always participate in the CA assessment process.

b) A poll was taken regarding the maximum percentage of filtered out assessments allowed to be a vCA: 1 - 10% - 0 votes 10 - 20 % - 4 votes 20 - 30 % - 3 votes 30 - 40 % - 2 votes 40 - 50 % - 0 votes 50 - 60 % - 0 votes

The notes from the full discussion are on the 19.1.2022 section of the Miro board: https://miro.com/app/board/uXjVOXVhuVk=/

[whitevo]

Video for 19.01.2022 discussion: https://youtu.be/TecEOUL7E_U

[kenricnelson]

These are notes put together for the Discussion with IOG regarding Fund 8 CA improvements on Jan 25, 2022. https://docs.google.com/document/d/1_hKY5HrU51WtlmlVhvLVps9oyhcU3dmw32JV9SFZU0g/edit?usp=sharing

And this the document describing IOGs action plan for Fund 8, 9 and beyond regarding the CA process. https://docs.google.com/document/d/1__RZNfCB3v-0jqOg1zQyStpgeL1hUaegGIhPhT-p8YI/edit?usp=sharing

This is the video recording of the Jan 25, 2022 meeting: https://mail.google.com/mail/u/0/#search/mike.mcnulty%40iohk.io?projector=1

[nadiahopkins]

Kenric and I discussed this prioritized problem and noted that, in Fund 8, a similarity analysis tool was implemented by IOG, and the CA guide was updated with expectations for assessing in line with this tool. The tool will serve an initial filtering function to flag assessments where there are specified integrity or quality issues. There is also a second filter in place following the vCA process to identify potential outliers, and a resolution process in place composed by the community to enable a feedback loop in such cases.

The updated CA Guide with explanation of the SA tool can be found here.

The new vCA guide with explanation of the resolution process can be found here. https://docs.google.com/document/d/1eSX455MCilIe247xRVi51UZLGfKGPBrxJthvtV5sASg/edit#heading=h.9ch3vgvwl4mz

The F8 vCA Eligibility and Incentives Requirements (Reputation) can be found here. https://docs.google.com/document/d/14qHSZ2uSHq2CZyeqju6KLBcGVfagesCLvavZMAgcbWo/edit

A recording of the vCA F8 Improvements After Town Hall can be viewed here: https://www.youtube.com/watch?v=EbzwP6ToKRs&t=26s