Add App Mining delivery instructions for monthly rankings from reviewers

[markmhendrickson]

We need to provide instructions for reviewers to submit their monthly rankings by completing the following steps:

• On first day of month, receive spreadsheet template from administrator via email from "app.mining@blockstack.org" with pre-made columns for "App Domain" and "Ranking" and all app domains from App.co pre-populated per https://github.com/blockstack/app.co/issues/210
• Fill out spreadsheet with current month's rankings (e.g. app domain "misthos.io" with ranking "17"), ensuring accuracy of domain and uniqueness of ranking
• Before 15th of month, reply to email from administrator, attaching spreadsheet and writing summary of rankings in body (including information such as description of how rankings were generated and observations about most notable rankings)

Note that reviewers do not have to rank all apps listed in the spreadsheet they are sent, but they cannot add any additional apps to the spreadsheet.

Apps will be identified by their domain during processing. As such, they should not be changed or reviewer submissions may be rejected for the month. App names are included in the spreadsheet simply for clarity.

Design: https://30a49b69e76343b8be73064ea33b99cc.codepen.website/app-mining/reviewer-instructions.html

Path: /mining/reviewer-instructions

[markmhendrickson]

@jeffdomke Assigning you this issue to get your feedback on the current set of instructions and how they map to an initial design proposal here:

https://30a49b69e76343b8be73064ea33b99cc.codepen.website/app-mining/reviewer-instructions.html

As with https://github.com/blockstack/app.co/issues/203#issuecomment-417631102, do you have a sense of where this may live publicly? Perhaps as a sub-path of /mining?

[stackatron]

Looks good to me. Subpath sounds right.

Step 2: Maybe clarify "The best, or highest rank app, will be #1, followed by #2, etc. (In case they try to pass us an arbitrary raw scoring or something).

One corner case, I wonder if they will discover apps that they feel shouldn't be ranked because they are scammy or don't actually use auth. There might be a difference between "I don't want to rank" vs. "this app should be removed from consideration". Anyhow, think we should ignore this for now.

[markmhendrickson]

The instructions have been updated with the addition of a line saying "For example, the best – or highest ranked app – should have value "1", followed by "2", etc." per your suggestion.

[moxiegirl]

• I think this benefits with a two simple topic sentences explaining what you are doing. The current version buries this information in the instructions. They make the instructions difficult to follow.
• Also, you should include a standard return Blockstack email here --- some people are confused by looking at email headers. And we should have a standard, leaving out the email implies the return can change.
• Move the example link to after how to rank information. That way the example can illustrate a ranking as well.
• If you send a spreadsheet, you should see about adding a function to the sheet that prevents users from entering a repeat ranking.
• The order of information around return is unclear. Put the warning about returning at the top.

---

Reviewer instructions for App Mining

Reviewers submit their app rankings to Blockstack every month. The first day of the month Blockstack will email reviewers a spreadsheet on which to record rankings.

To submit app rankings, you complete the following steps:

1. Download spreadsheet for that month.

   The spreadsheet is sent to the email address we have on file for you. The sheet lists all apps eligible for ranking.

2. Record your rankings on the spreadsheet.

   Place a the rank value in the Ranking column next to each app you would like to rank. You can rank only those apps you want and omit ranking others. Rank each application numerically where 1 (one) is the highest.

   Please avoid modifying the spreadsheet in apart from adding your rankings . The App ID column, in particular, must be preserved to process your rankings accurately.

   View an example of a completed spreadsheet.

3. Return your rankings by the 15th to email@blockstack.org

   IMPORTANT: We must receive your rankings by the 15th of the month or they won't be considered!

   Attach your completed spreadsheet to your email. Include in the email a brief explanation summarizing your methodology or thoughts when ranking. Please mention any app you want to highlight for the month.

[moxiegirl]

You may want to include an example email --- screen grab is fine. That way reviewers can get an idea of what a summary should contain and how long it should be. Also, I assume that app developers will be able to read these instructions so they can see in the examples how ranking and commenting is done.

[markmhendrickson]

@moxiegirl Thanks for this feedback! It all sounds good to me. I'll incorporate it into the design soon and ping you again to take another look.

Per "you should see about adding a function to the sheet that prevents users from entering a repeat ranking" in particular – I think we're going to send them a simple CSV file so I'm not too confident we can layer in macros for this.

But @hstove do you think there's a possibility of supporting this functionality while maintaining compatibility across spreadsheet apps somehow?

[markmhendrickson]

@GinaAbrams @pstan26 do or can we have app.mining@blockstack.com set up for this process? Or do you want to use a different address (such as one of your individual ones)?

[markmhendrickson]

@moxiegirl I've updated the designs here with your suggestions, just haven't included a screenshot of the email we send out yet.

@pstan26 @GinaAbrams want to draft out an email in your language that serves the purpose of step 1 here and provide it here (both subject and body) so we can include it as a screenshot?