Reviewer: App Awareness

[pstan26]

Right now it is too easy to game a metric like daily active users, and apps are not currently incentivized to grow their user base. A solution could be to add an App Reviewer that measures an app's awareness by reach.

[pstan26]

original post here: https://forum.blockstack.org/t/propose-an-app-reviewer/6660/5

[pstan26]

Currently in talks with Awario. They feel confident they could remove gaming of this system to an acceptable degree by evaluating true "reach" of an app over "mentions" (which are easier to game). They seem interested in moving forward as an App Reviewer, would like to compare others though.

[pstan26]

Also, I like Friedger's idea that "Maybe have the app review consists of one score from centralized services and one from decentralized services and over time shift the weights towards the latter." Starting 100% in the former camp sounds reasonable however at this stage.

[fiatexodus]

@pstan26 If your initial suggestion evolved to something like measuring recurring active user growth, that would be useful. If there was a decentralized analytics company you could rely upon for all blockstack apps that would be ideal.

Reach and mentions concern me because they both seem easy to game and don't really measure the true utility of an App, just the founder's social media and PR prowess.

Perhaps you are concerned that it would be easy for people to game recurring active user growth though--without grabbing identifying information in the analytic?

[cuevasm]

Hey, good thoughts, wanted to add some context here since I use Awario regularly. The raw mentions number could certainly be gamed (just have a bot tweet out your company name a bunch), however the reach is much harder to game and that's why we plan to focus on that as the key metric. Gaming this in any effective way would require getting large legitimate accounts that manage to escape Awario's filtering to mention your brand. Not only do they do a great job of assessing for accounts that are real, they also offer blacklisting tools so that if ever there was an account that was set up to game the system AND it managed to amass a bunch of followers/potential reach, we could very easily remove it from the counts. Furthermore, the idea is that all the mention and reach data will be available for everyone to look at and help us identify these likely rare cases and this would be in addition to audits by Awario and Blockstack PBC

More generally speaking, it is necessary to market any product or service if it is to be successful . Especially in the crypto world it seems, makers should be thinking about this earlier on. App Mining should encourage things that are proven to set you up for success. Users can't be acquired without awareness. I do agree that this type of App Reviewer gets even stronger when/if a true user count is introduced (my opinion is that it should). The two scores could work together to reward something like high conversion to eliminate any cases where an app is able to generate a ton of awareness but really low usage (probably unlikely but possible if they are marketing way way outside of their market). In the meantime though, it's important for apps to build their awareness and expose their offering to potential users.

[cuevasm]

We're going to proceed with Awario but do a dry run - this will give everyone a chance to see what type of data we'll be getting and how it would affect scores before it becomes actual. Ongoing is work to finalize how to normalize the score.

Open question: High-level, do you think that we should reward recent growth in this area fairly heavily? i.e. instead of focusing strictly on volume, it would look at the growth compared to last period. So, there's this notion of momentum and it's a bit more relative across apps.

[hstove]

Related to Mitchell's last post, we've been discussing potential algorithms for use with Awario. Here is my proposal:

Reach Score: Based on total reach. Your 'score' is log10(total_reach). So if you've reached 10 people, your score is 1. 100 is 2, 1000 is 3, and so on. This is much better than only using your actual reach, because outliers would totally skew the distribution. No matter what your reach is, you need to improve 10x your reach to increase this by 1. Using log10 is also similar to how we handle the 'theta' function in the algorithm, because the higher your score, the more you need to improve to bump your score.

Growth Score: Month-over-month growth in your total reach (not log 10). If you went from a reach of 1000 to 1500, your MoM growth is 0.5 (or 50%). If this is your first month in app mining, this score is not included.

Like all the other reviewers, we first calculate the z-score for each of these metrics, and then average your z-scores. Then we apply the theta function, and you have your 'final' Awario score.

This is just my proposal, and is not final.

[cuevasm]

Definitely makes sense to me. I like how making it log10 and including the growth score month over month further reduces gameability and controls for any small amount of error we might have in picking up reach count that wasn't actually about the project. It is also positive that newer applications will be able to compete even with well-established projects.

[stackatron]

@hstove @pstan26 @cuevasm we should have a chat about finalizing this plan.

[avthars]

Love this idea. We should def incorporate this!

[cuevasm]

Everything is now in place to do the dry-run with Awario. Today we sent over a list of all current apps and the Awario team will be adding them to the tracking system. After enough data has been collected I'll post here! An announcement should be coming soon as well, we'll be working with their team on the exact rollout there. The plan is to proceed with Hank's proposal on the scoring methodology for the dry-run and then take feedback after having that to look at. Post here with any questions or concerns, thanks!

[cuevasm]

Hey Miners! Excited to introduce Awario. Here's the full announcement and remember, this first month will not count, we'll provide data and have a chance to review and implement feedback before they become official. Take a look at their announcement as well!

Details on scoring:

First things first, as you dive in here, I highly recommend checking out Awario’s docs, it should answer most specific questions about how the platform works - https://awario.com/help/.

Second, please note that Awario is finishing up work on a way for you to easily login to the interface and run your own reports, slice and dice data, see your Mentions, etc. In the interim we’ll be providing these manually via PDF and CSV, but we don’t anticipate that being the case for more than a month or two. Part of our agreement with Awario is to provide this feature as we felt it was really important for you to have direct access so that a) everything is transparent as possible and b) the information is accessible and actionable to you and your team.

With that, here’s the how we’re proposing the score to work and how it will be done for the dry-run. There will be opportunity to provide feedback before it becomes a part of your official score.

The Awareness Score

At a high-level, Awario focuses on two major aspects of awareness:

• Mentions: These are captured mentions of a brand or app online, on social networks, and on news sites. A ‘mention’ is registered when the name the brand or app appears publicly. Example: A tweet mentioning the app name.
• Reach: The estimated reach of the combined mentions collected for your brand. Example: How far the tweet about said app traveled online.

For the purposes of App Mining, the focus will be on Reach. Mentions themselves will be captured and provided to App Miners, but Reach is the much less gameable of these two numbers and thus more suitable for App Mining. For example, it would be fairly easy to create many fake individual Mentions (e.g a Twitter bot), but it would be unlikely those ‘fake’ Mentions generate much if any actual Reach.

Here’s how the scoring will work in more detail:

Reach Score: Based on total reach of all your eligible Mentions for the previous calendar month. Your 'score' is log10(total_reach). So if you've reached 10 people, your score is 1. 100 is 2, 1000 is 3, and so on. This is much better than only using your actual reach, because outliers would totally skew the distribution. No matter what your reach is, you need to improve 10x your reach to increase this by 1. Using log10 is also similar to how we handle the 'theta' function in the algorithm, because the higher your score, the more you need to improve to bump your score.

Growth Score: Month-over-month growth in your total reach (not log 10). If you went from a reach of 1000 to 1500, your MoM growth is 0.5 (or 50%). If this is your first month in app mining, this score is not included.

Some more conversation about why this way is above as well.

Other scoring notes:

• The query and search parameters for ‘mention alerts’ are really important and this is why we have the Awario team handling it directly using their expertise on their own platform. For unique brand or app names, the query is fairly straightforward, the name is loaded into Awario and it begins crawling sites and networks for public instances of it. For common names or names where context is important, such as a brand name like Stealthy (hey guys!), it’s important to filter out non-relevant results like someone simply using the verb ‘stealthy’. Agora is another good example as there are other projects with the same name. For these cases, Awario sets up much more complex queries to trim it down to the ones actually related to the project. Here are some details on all the operators. Additionally the first month there is no score as the Awario will be honing that score by setting up a first version of the mention alert, watching it, and then updating as necessary to further zero in on only relevant mentions. We will have the Awario team here and on calls to answer more detailed questions on this should you have them.
• App Miners will not receive a score from Awario in their first eligible run, as mentioned above this is because setting up the Mention queries and optimizing them to capture only relevant Mentions takes some time and they should be finely tuned before counting them.
• ‘Websites’ are excluded from Reach totals. While this is generally pretty accurate and useful in a normal business use-case, it can be gamed because of the way Awario estimates the Reach of website. The sites Alexa rank is used in the Reach calculation meaning, for example, a mention on Github would register as massive Reach, even if that particular Mention didn’t really spread that far.
• Like all the other reviewers, the z-score is first calculated for each of these metrics, and then then averaged. Then, the theta function is applied, resulting in a 'final' Awario score.
• Mentions and associated ‘Reach’ from Blockstack accounts will not be counted. This is so that Blockstack PBC can continue to support apps publicly, without worrying that it needs to be evenly distributed across apps, which simply isn’t possible.

Auditing As part of the monthly process for generating these scores, the query will not only be optimized for mentions of the eligible apps, Awario will also help to audit the Mentions coming in to be doubly sure none that shouldn’t count make their way in. Awario is confident in their ability to only collect and count relevant Mentions through their search operators and will be available to answer and questions or concerns App Miners may have. Last, with Awario’s platform, it is extremely easy to remove individual Mentions (and thus the associated Reach count) or to blacklist any accounts found to be fraudulent or accounting for false-positive Mentions.

[cuevasm]

An additional scoring note, we'll also be restricting the handles of all Blockstack PBC employees, meaning any mention and accompanying reach generated by a PBC employee will not count in anyone's score. This is so everyone at Blockstack PBC can continue to freely support applications without running the risk of unintentionally biasing the results or deciding not to support publicly for fear of impacting the results.

[cuevasm]

Notes about Awario Scores for April

Basics:

• Your score is comprised of data captured from March 1st to March 31st, 2019
• You’re getting two reports
  • Raw CSV of all Mentions we’ve captured to date (more on this below)
  • Mention/Reach/Influencer/etc. Summary PDFs (nicely formatted report including additional data about who spoke about you and how)
• If there’s no spreadsheet or PDF linked, there was no data for you yet, go get some Mentions! :)
• Reminder, this is how scoring works for Awario

Materials:

• Awario Data Sheet
• Awario Scoring Sheet (see you how Awario scores would have impacted you in April
• Folder of all App Mention CSVs (all data captured so far, not restricted to scoring window)
• Folder of all Report Summary PDFs

Important Notes:

• Mentions from Blockstack PBC employees will not count toward your scores, we’re in the process of blacklisting these accounts. This is so the team can continue supporting Blockstack apps without fear of biasing the results. This is the same rule we applied to official Blockstack accounts like @blockstack. Please note: You will see mentions from the team included in the scores for this month only - since it is not impacting payouts, we prioritized other items, but will be sure to have those out before next month. And yes, they’ll be removed from the month previous (this month’s data) so growth calculations will still be accurate.
• I’ve made your reports accessible to anyone with the link. This was for ease of sharing and so everyone can look at all the data (and there’s no personal data being exposed). If this is an issue, please let me know. This won’t be a problem in the coming months as we’ll have a direct login to Awario for you to use so you can access it yourself, allowing you to slice and dice in the way that works the best for you and generate any reports you need.
• You will not have a ‘Growth’ score this month since this is the first month capturing data. The same will happen to every new app as described here. Please remember we excluded ‘Web’ as described in the scoring breakdown.
• Reporting a Mention. If you’re looking through your data or someone else’s and see a Mention that isn’t about the app, you can let us know by sending us the URL available in the CSV. We’ll investigate and remove if it’s a false positive. Again, as we move forward, this will become a lot easier as we’ll be able to easily browse through Mentions right in Awario and likely use the ‘star’ function in the interface to mark mentions you believe are in error. We anticipate these to be pretty minimal and for any that do sneak through to be pretty inconsequential in the overall scoring picture, but it’ll be great to have us all watching in case there are outlier cases.
• I’ve left Web mentions in your export of Mentions AND given you all the Mentions we’ve collected thus far (not just the March ones). Even though not all of these are included in your score, they’re still really helpful data and it may show you some nice Mentions you missed (I recommend following up to see if they will talk about you more, work with you somehow, or otherwise boost your app somehow!). With this data, you can filter the Sheet to the current date period (March 1-31) and exclude websites if you want to look at what comprises your current month’s score.

[polluterofminds]

I’m still trying to wrap my head around how the reach score works. If reach is based on mentions, can someone define mentions a little clearer for me? I read all this page and the Awario help docs and I’m lead to believe that a mention is anything by someone else mentioning the brand. This would exclude, for example, tweets from the brand that grow legs and have a high reach number. Is that right? Or am I misunderstanding this?

[cuevasm]

Hey Justin, I would reference this --> https://docs.blockstack.org/develop/app-reviewers.html#awareness-scoring

Mentions in our case is a Mention by anyone (excluding those by Blockstack PBC employees or official properties). In a normal business use case, Awario automatically excludes Mentions from your own brand accounts. This is because you should know about those and you're hoping to capture and look at what others are saying.

In the case of App Mining, we don't exclude your 'self Mentions'. A major point of Awario as App Reviewer is to encourage brand building and content sharing by App Miners. So, excluding them would mean not counting the hard work everyone is doing to build their brand and get their content out. At this stage, most are probably better suited to go find influencers and have them tweet about their App, but we didn't want to discount the other activity in the meantime - I personally don't see a reason to do that, but am all ears if you think we shouldn't reward this behavior.

I'm not sure how to make Reach clearer but I'll give it a go--it's just the estimation of how many people saw your Mentions. Different networks estimate a reach of a Mention differently and some of this proprietary, but most of it is pretty obvious, aka a tweet by someone with a few hundred followers is estimated to Reach a few hundred. The imperfection in these estimations is something we've addressed, the summary is that 1) everyone is measured the same way so the same imperfections apply and 2) the score is logarithmic so even if at times, the Reach is skewed a bit, it won't have too big an impact on the end result. We also audit for anomalies and will discard Mentions with obviously wacky Reach scores if they come up (hasn't happened yet). We've also excluded Websites because these estimations are not as reliable as the other places Awario is combing.

[dantrevino]

It appears that some are just buying tweets and that's not a good metric for measuring value.

[cuevasm]

Any tweets you suspect of this should be reported, it's very easy to blacklist accounts we can show are spam or bots.

[polluterofminds]

Thanks! I didn’t think to look at the Blockstack docs. You mentioning your own brand should absolutely be included. I was just worried that it wasn’t.

[andresousa]

If we think some tweets were bought, where should we report them? Here on this thread?

[polluterofminds]

To piggy back on @andresousa's question, should tweets be reported if they appear to be bought or only if they are spam or bots? There's a difference. There's a marketplace of real people willing to tweet to their hundreds of thousands of (probably fake) followers for a price. So, is that spam? Should tweets that seem to fall into that category be reported?

I would imagine Blockstack wants to see real reach for apps on their platform, so when someone pays a person to tweet, that's not real reach. In fact, engagement is probably the more important metric than reach anyway, but I digress.

[andresousa]

Agreed @jehunter5811 ! My first question should have been "should tweets be reported if they appear to be bought or only if they are spam or bots?".

[jackveiga]

After looking at some of the awario results I've noticed a pattern in two apps at least where they have twitter accounts tweeting or RTing repeatedly the same tweet literally over and over (these accounts have a decent following but from the looks of it, it's all purchased tweets). Some of the mentioned apps don't even have any activity on their app official twitter account for the past month, they don't even make the effort.

If awario can't spot this then it's fair to say that the statement that awario feel confident they could remove gaming of this system to an acceptable degree is not working out.

I'm not against paid tweets or paid marketing, it's a strategy. But either the system needs to be able to catch these apps that abuse with strategies like paying account to post repetitive tweets/RTs and discard them, or alternatively, the app awareness needs to be ranked in a combination of factors to dissuade these strategies of using paid spammers.

We tried hard with recall on platforms like twitter, from going after people that mentioned privacy or google photos and engaging with them directly, setting up for twitter ads to get an extended reach and spread the word in threads where there is an active discussion. We also noticed that awario most of the times don't count any of these interactions according to our mention report.

Lastly, reddit is being unaccounted for by awario (at least in this latest report) because it's easily gamed but sucks to see recall mentioned on some reddit threads with multiple comments and upvotes, that pop up from the time to time on privacy subreddits that leads to actual installs and blockstack awareness not counting in the slightest.

[pstan26]

@jackveiga good review. One thought I've been having is that it may be difficult to stop people paying for posts, but I wonder if its possible to just weight people's follower to following ratio who tweet about an app, assuming higher ratios mean higher signal and less spam. So if someone has 1M followers but only follows 300 people, they would be weighted more than someone who has 1M followers and follows 1M people, and more than someone with 1000 followers and follows 1 person.

[cuevasm]

I think this is something for the community to ultimately decide on. From Awario's point of view, buying tweets isn't gaming anything, this is effectively creating Reach online. Many forms of Reach are ultimately paid for, on Twitter it's called influencer marketing and it's very effective.

I haven't seen super strong evidence of bots or spam situations, but I think I would generally be against them (without banning good forms of automation) and Awario has built-in protection from suspicious accounts already. RE: Patrick's suggestion on signal, I think it's a fine idea but would require Awario building this in. I think sometimes follower ratio can be an indicator of quality, but isn't always necessarily.

Reddit is counted and any Mentions for you there would be included - I see one in your data for May, remember that mentions are from May 1-31, if you see Mentions on reddit that aren't in your sheet, let us know, there may be a reason they weren't picked up or counted @jackveiga

Last, I'm not sure why this matters @jackveiga: "Some of the mentioned apps don't even have any activity on their app official twitter account for the past month, they don't even make the effort." Just because they have decided not to emphasize their twitter account doesn't mean they aren't putting in effort in other places to be visible. They may have decided the best strategy for them is to not build their own account, but get others to talk about them. Effort and using one's twitter account are not necessarily related. As for paying for tweets, again, I think this is something the community can decide on, paid marketing of many forms is something almost all businesses do and I don't think we can say Awario is failing at detecting gaming in this situation.

[polluterofminds]

"Many forms of Reach are ultimately paid for, on Twitter it's called influencer marketing and it's very effective."

Paying to promote a tweet is effective. Paying someone with 900,000 followers (who mainly may be bots) to boost your "reach" is not effective. The definition of effective, of course, is subjective here as well. Any reach is good reach for Blockstack. For apps that actually want to have a company after App Mining ends, then engagement matters about 1000% more than reach. Reach doesn't necessarily convert to anything other than noise.

Listen, I'm good with whatever everyone decides, but it'd be a shame if Blockstack encouraged the same shady growth hacking that silicon valley already does. We have enough of that in tech and it'd be cool if the change makers actually looked for ways to do things differently.

[cuevasm]

Yep, I'm saying we should decide for ourselves - it's going to be really tough to separate out the two at scale without punishing some forms of good marketing. Effective might mean bootstrapping your brand with paid engagements until you catch real steam or until you get in front of the right audience or journalist, etc. so it's hard to say for everyone, 'no you shouldn't do that or you'll be punished for it, even though this is something all your centralized competition is doing'.

And I would stop short of saying that not explicitly banning something is encouraging it - I wonder if the best case here is to create an App Miner's code of conduct and we feel certain things go against it, we flag them and then the community votes to decide if it should be something fair to competition or not. Ideally in those cases the Miner doing the activity in question would be willing to stop if the community votes if down (without us having to put in new technology to stop 'cheating'), they would adhere to the code of conduct because they want honor the community decision and ultimately remain part of the program.

[kkomaz]

Is the TLDR consensus that paying for twitter bots AT THIS TIME is ✅?

If not who do we report this to?

[jyudkin1]

Going to comment here -

What is the difference between paid ads = giving Facebook / Twitter / Instagram $$$ to reach users. Many of which are potentially bots / fake accounts etc ... Lets assume this is priced into the market. I would assume most people in this space are aware of this.

Vs paid Influencers? Independents who have built audiences (loaded term) bypassing the duopoly gate keepers.

Vs native ads / placements?

I believe there was a slack conversations about this topic as well.

Second - This would be an activity that any and all businesses would build. Whether you depend on word of mouth ("organic" - natural), paying Facebook, Google, Twitter etc for traditional digital, influencers, Experiential (in person activations), SEO (google etc), SEM, OOH, Television, Radio, Streaming, etc.

The app mining program generates rewards for developers. BTC can be deployed to hire, resources, reach more customers, awareness, PR, content creation, Social mgmt, conference attendance, community building etc.

All of these activities should ideally grow your user base, increase awareness of an app, and ideally grow the ecosystem as a whole.

[cuevasm]

Hey all, I think this is getting into separate issues and is getting away from the original thread (plus it's a closed issue and this isn't a closed discussion). I've started a thread explicitly to discuss paid vs. not paid marketing activities, etc. Please join me over there --> https://github.com/blockstack/app-mining/issues/117