ripcurlx
commented
6 years ago Thanks to @ManfredKarrer and @cbeams for giving feedback upfront on the first draft of metrics!
Open ripcurlx opened 6 years ago
ripcurlx
commented
6 years ago Thanks to @ManfredKarrer and @cbeams for giving feedback upfront on the first draft of metrics!
ripcurlx
commented
6 years ago Based on this trackings/event log we also would be able to see:
Besides this standard KPIs, we also could think about metrics that would represent the core (privacy, trust, security) of our product (credits to @cbeams and @ManfredKarrer) as:
As always if you have any questions on any of this, feel free to chat with us on https://bisq.network/slack-invites - #analytics channel. :smile:
ripcurlx
commented
6 years ago After discussing with @mrosseel I'd like to go into further detail on the events we should collect to have actionable metrics. Every event should, if possible, have a distinct identifier (e.g. distinct_id) so it is possible to see at what point of the app users have problems with. Without a distinct_id it is not possible for us to differentiate between events sent from different users. Of course the generation of the identifier (e.g. hashing of onion address with salt, that is only known to user) has to be secure and protect the privacy of the user. User should be able to opt-out of this at anytime or to change the salt of the identifier. Additionally to the distinct_id a timestamp has to be stored for every event to be able to monitor events over time. Last but not least we'll need to store the app version for each event to differentiate behavior between app versions
| Event | Properties |
|---|---|
| Started App | |
| Created Offer | {"type":"Sell\|Buy","amount":"0.25"} |
| Accepted Offer | {"type":"Sell\|Buy", "amount":"0.25"} |
| Completed Trade | {"type":"Sell\|Buy", "amount":"0.25"} |
| Reviewed App | {"channel":"cryptocompare.com"} |
| Submitted Feedback | {"type":"positive\|negative", "message":"I love Bisq!"} |
| Stopped App |
Following events can be collected without connecting a user id for drawing conclusions
| Event | Properties |
|---|---|
| Downloaded App | {"app_version":"v0.6.0"} |
| Updated App | {"app_version":"v0.6.0"} |
Topics to discuss:
Learning Goal
To be able to tell if changes in a new version of Bisq were a success or failure we first have to define the meaning behind these words. Defining success as an increase in volume probably has nothing to do with the actual changes, but more with the current market situation of the cryptocurrency space or just plain with public holidays in our main markets. The goal of setting up KPIs is to give everyone a tool to be able to measure the success of any change on the platform, find out problems more quickly and to be able to tackle issues we may not be aware of at the moment at all.
This issue should be a starting point of a discussion on what and how we want to measure our success.
Hypothesis
If we track following KPIs [1] and make it accessible for everyone then everyone should be able to see if a change to the platform was a success or a failure. This should cover different parts of the client/network without being changed for every feature we roll out.
[1] KPIs: To be able to differentiate between different versions of the client we need to be able to segment every metric we collect by version number of the client. Additionally we want to collect the data on a daily/hourly basis to be able to make assumptions about external incidents that occur. To be able to measure conversion rates within the client we could collect data for specific onion addresses and store it by hashing the address with a salt.
Metric
This quantitative metrics we are collecting could give us e.g. following actionable insights:
Experiment
The collected metrics should be accessible over an API with a JSON response that can be used e.g. in a Google Spreadsheet or other analytics tools.
Result
Next Step