Wallet Scrutiny - Project Exploration Call

[sbddesign]

Date: 2022-10-27
Time: 7:45pm
Timezone: UTC
Duration: 1h
UTCTime: 2022-10-27 19:45 UTC

Agenda

We'll talk with the Wallet Scrutiny team about UX/UI improvements to their website. Here is Tanner's original post in Slack.

In short, this project is focused around testing which bitcoin wallets can be independently verified against their source code. They plan to serve 3 different types of users: end users of wallets, security experts, and wallet devs.

Check your timezone

https://everytimezone.com/s/4de1d4c7

Join the call

https://meet.jit.si/bitcoindesign

Calendar invite

Subscribe to the Bitcoin Design calendar.

[johnsBeharry]

I think with the things coming into bitcoin, community banking, fedimints, taro tokens, this kind of scrutiny website is very very necessary for us to continue having a healthy ecosystem.

[Giszmo]

Hi guys, founder of WalletScrutiny here.

So the current design was done by @vswee as a surprise PR at some point and he has helped us ever since (and also gave this issue a :+1:) but other than that, the project had much more contributions from technical people like myself, so any help and Tanner's reach out to this community is much appreciated.

Apart from what you might already find non-optimal in the current website, I think the main pain points for the project are:

• Trust: While the main theme of the project is to have all aspects out in the open and subject to scrutiny themselves, we do make errors and occasionally disagree with product maintainers about the presentation of their products. This leads to accusations being thrown at us on social media, in the worst case by a provider that has us all blocked on social media, so our counter-points - if we get aware of the drama in the first place - won't get heard where they are being expressed while at the same time the provider never opened issues on on our issue tracker where we could have resolved such issues long ago. Therefore I think it is crucial to improve the community engagement such that disputed verdicts maybe get shown as disputed with a discussion linked where both sides get to state their case in a way that is resistant to censorship from any of the stakeholders. Only with absolute transparency and criticism being linked can judgements or ratings be trusted. Keeping drama off social media would also free considerable resources to focus on the project.
• Community: So far, all verdicts went through my approval and 20 pull requests with probably 200 additional verdicts are pending my review. I'm not happy with me being that central authority and bottle neck and have much hope to get more wallet devs or security experts to help out. One thought was to have providers categorize and test the products of other providers maybe to gain credits for their own products. Currently there is only a semi-meaningful sorting by users/ratings for Android and iPhone wallets beyond the verdict. The sorting of hardware wallets and bearer tokens (and desktop wallets in the future) is random within each verdict. Credits earned this way could be rewarded with a favourable sorting. If it fails to get much contributions it could back-fire though.
• Nuance: The project is mistaken for a ranking by security. "Green is secure." I try hard to communicate that transparency doesn't mean that anybody actually cares to look at the code, so the most transparent wallet might still lose the users' funds easily. I worked for a wallet with a million downloads on Play Store and in five years there was less than a hand full of comments from external security researchers on our public source. I am convinced that with the exception of Bitcoin Core, any wallet could pull off an exit scam as it costs probably more to audit every release than to implement them if deliberate back doors should be prevented with high confidence. In that sense, we are overselling "reproducibility" - the only green verdict - as the pinnacle of security while it's only a necessary property for being audited in a meaningful way.

Three years ago, when I started the project, my vision was that we would have a crowded top category giving us the luxury of creating new top categories ("audited" would be nice but that, too would be easy to fake. Who audits the auditors?) but given the reality that there is not much of a trend towards wallets getting reproducible, I now think it's important to get to the security aspect by other means. Instead of a binary - audited? Yes or No? - I'd like to add "expert opinions" that get loaded from an external source and are free from our editorial influence. How to implement and present those will be a design challenge.

[sbddesign]

We covered a lot of ground on this call. We can continue discussing in the #wallet-scrutiny Slack channel.

Videos:

• YouTube
• BitcoinTV

Some of my notes (not the cleanest to read):

• How to avoid editorial bias?
• Wallet users
  • UX exploration
  • How do they choose a wallet?
  • What's the correct amount of information to show them?
  • 10 point scale?
  • Past rating system? (Last time failed, last time passed, etc)
• Wallet Developers
  • Some have been critical, most are not critical but simply not engaged
  • How do we improve relationships with wallet devs?
  • How to be friendly with wallet devs without compromising principles of website?
• Security Experts
  • Miracle for few people on call to have covered so many wallets
  • Some wallets cant be reproduced
  • Some can only be reproduced by the wallet devs
  • More experts involved will helped with volume and expertise required to get accurate infiormation
• Research studies
  • Around wallet users
  • Around wallet developers
• How will this info be surfaced in future?
  • Data scraped directly into the app store?
  • WalletScrutiny badge?
• "Don't want to be simplified to where it becomes an opinion."
• Different buckets:
  • UX
  • PR/Education
  • Contribution
• Competitive Research
  • https://reproducible-builds.org/
  • https://bitcoinbinary.org/
  • Look to non-bitcoin stuff that involves rating and reviews (think Yelp or Google Maps)
  • Again, don't want WS to drift into the realm of opinion, but look at how these services present information to the user in a way that's useful and intuitive
• Accredited experts - 1 developer per wallet project
  • They cant just downvote all other wallets because others could do same, so it would be mutually assured destruction?
• Rating a wallet not as simple as a clean or yes or no
  • Some wallets have different risk factors
  • Some risks are subjective, not objective
• Potential Next Steps
  • Research
    • Other reproducible build
    • Review and rating sites
  • UI design
  • Marketing and PR
    • Copy drafts