[Feature] Improvements to BSQ supply charts and figures

This is a suggestion to make some improvements to the BSQ supply charts and figures that are found under DAO > Facts & Figures > BSQ Supply. For a quick and cheap introduction, at the bottom of the post there are two screenshots with captions.

Unresolved issue threads that talk about improvements to BSQ facts & figures panels: largely talk about how BSQ burn statistics are presented:

[Feature] Provide a way to show burned BSQ in a specific timeframe #3303
Improve burned BSQ chart #2990
Show BSQ trade fee in % per day #3292

I decided to open a separate thread, because this proposal has a slightly larger scope than either of those.

Suggestions

use stacked area charts, because both charts in the "BSQ Supply" panel are already presented as sum totals of a few series (e.g. series in first chart are BSQ issued at genesis tx + BSQ issued for compensation + BSQ issued for reimbursement);
in the "BSQ issued" chart increase detail with shorter intervals (currently the interval is 1 month);
offer charts and figures for multiple time periods (more on that below);
find a solution for outlier resilience (more on that below);
consider using moving averages to improve chart readability (more on that below).

The last three suggestions are less clear-cut or obvious, so details below.

Multiple time periods

That's multiple time periods in the most general sense. It can either be the ability to select from a small set of time periods (last day, last 12 months, etc., like in Market > Trades) or it could be a way to specify a custom period (like a slider). I see two motivations for multiple time periods:

comfortabily examining the series on different timeframes; and,
seeing the aggregates (e.g. total BSQ issued) for different timeframes.

I think the optimal user experience and development effort would come from the simplest solution that works. So if a set of 3-4 timeperiods is enough (like in Market > Trades), that's preferable to a slider or some other "more interactive" solution.

Outlier resilience

Recently an outlier (point outside multiple standard deviations) appeared on the BSQ burnt chart (as seen in the screenshot at the bottom of the post), rendering the rest of the chart unreadably flat. Ways to offer outlier resilience:

automatic explicit visual trimming of outliers (outliers drawn outside the chart, and thus invisible and not distorting);
optional switch to logarithmic scale;
ability to select which portion of y-scale to zoom in on;

I think automatic explicit visual trimming of outliers is by far the most user-friendly method and the easiest to implement.

Moving averages

A moving average can make trends in a noisy chart easy to identify, but it can also be visual clutter if the series is smooth or you don't care about the trends it captures. I think MAs are something to keep in mind if there's a need to examine the trends. I am not familiar with the analysis usually done using these charts, so I can't speak to that.

Do you have a use case for these facts & figures?

Above take on the problem is probably not ideal. Since I am not very familiar with the use cases (user stories) for these charts and figures, it would be great if you could provide some in the comments. Some of the things use cases might help clear up:

What timeframes are relevant?
Are timeframes whose right-most edge is in the past relevant? E.g. [September to October], as opposed to [1 month].
What intervals are useful? Hourly, daily, weekly, monthly, etc.
How do you use (or would like to be able to use) the figures/aggregates?
Would there be a use case for a crosshair or tooltip that would appear when mouse hovers on a data point that would display/highlight the XY values of that data point?

PS

This bug report, that was closed due to inactivity, could be kept in mind while working on this:

Missing trade(s) in the DAO/Facts&Figures/Dashboard tab #2946

Figures

BSQ burned chart Burned BSQ chart. Notice the outlier on the far right.

BSQ issued chart Issued BSQ chart. The three series corresponding to the three figures above the chart could be visualized using a stacked area chart. Also, a finer interval could be used to make the chart more detailed.

If there's agreement that these are worthwhile changes, I'd like to call dibs on implementation.

in the "BSQ issued" chart increase detail with shorter intervals (currently the interval is 1 month);

Even if we change the detail it might not change the presentation a lot as BSQ are issued in a roughly 30 day interval after each cycle.

Multiple time periods

👍 - yes that would be good addition to this chart.

Outlier resilience

Recently an outlier (point outside multiple standard deviations) appeared on the BSQ burnt chart (as seen in the screenshot at the bottom of the post), rendering the rest of the chart unreadably flat. Ways to offer outlier resilience:

👍 - we need to fix this as we'll see this kind of outliers every now and then when the donation address holder is burning the acquired BSQ. I guess your first suggestion (automatic explicit visual trimming) would be a good start.

Do you have a use case for these facts & figures?

Above take on the problem is probably not ideal. Since I am not very familiar with the use cases (user stories) for these charts and figures, it would be great if you could provide some in the comments. Some of the things use cases might help clear up:

In general those charts should give developers and other future BSQ holders an idea on the BSQ economy. Is the project already sustainable? What is the trend? How did the BSQ price change in the past?

What timeframes are relevant?

I think as we have the 30 day cycle length for issuing new BSQ a timeframe that is bigger than one month should be possible to select. Comparing issued and burnt on a daily basis might be not so common as it will be only burnt BSQ for most of the time and only once within 30 days there will also be BSQ issued. So a monthly trend for me personally would be the most important one.

Are timeframes whose right-most edge is in the past relevant? E.g. [September to October], as opposed to [1 month].

You mean being able to select the full month of September and full month of October?

What intervals are useful? Hourly, daily, weekly, monthly, etc.

I don't think we need intervals < daily IMO

How do you use (or would like to be able to use) the figures/aggregates?

For me the major use case would be to compare BSQ burnt and BSQ issued over time to see if we are on the right track to sustainability.

Would there be a use case for a crosshair or tooltip that would appear when mouse hovers on a data point that would display/highlight the XY values of that data point?

I think a tooltip would be nice especially if we are using stacked charts.

But in general I think it is great to improve this section of the client. It was mainly a quick shot from my side when we released the DAO to have something to present to the user.

Are timeframes whose right-most edge is in the past relevant? E.g. [September to October], as opposed to [1 month].

You mean being able to select the full month of September and full month of October?

Yes. If we just need "last n months" type of periods the UI is much simpler than if we need "datetime x to datetime y" type of periods (where the end of the period isn't always the "now" datetime).

The input will be helpful, thanks!

Order of implementation currently is: 1) outlier resistance, to make the chart readable, 2) multiple timeframes, 3) stacked area charts, 4) way to see data point values (probably tooltip).

Outlier resilience

The burned BSQ by the burning man can always be an "outlier" but a very important one to see. Not sure whats the best option here, but filtering that away wont be a good idea. With more time line control this problem can be solved as well.

@chimp1984 the solution I'm currently finishing up on is to automatically recalculate the appropriate range for the chart axes, without taking the outliers into account. The outliers are not removed, but they are not captured whole in the viewport, and you can see that there's a point or points "outside" the viewport, because of how the chart line moves. Furthermore, the idea is to ultimately have a tooltip or something similar to be able to check the exact values of each point, so you'd be able to check the Y value by hovering on the X value, even if the Y value is not in the axis. I hope my explanation is clear enough. I can't provide a screenshot right now.

more time line control

If solving this with a control, I think a Y axis control, that would let you specify the range of points you're interested in, would be more effective.

That said, this does bring up an important point. Since starting to work on this I've learned that JavaFX charts are severely lacking in features. For example, if auto-rangeing is disabled, JavaFX axis don't support automatic tick-unit computation, I had to reimplement that. Also, JavaFX axis don't support a logarithmic scale, which is significant [0]. Also, little support for interaction, like panning or zooming. There are many other features that we've come to expect from charting libraries that are not there in JavaFX.

I brought this up on Keybase chat, asking if it would be possible to include a JavaFX extension library, like extjfx, which provides these features, but I was told that new JAR dependencies are not welcome.

[0] I'd like to expand on why I said that not having log scale support is significant: BSQ burnt over time is logarithmically distributed. Even after clipping out the outliers, it doesn't look "right" on linear scale. I guess it's up for debate if that's detrimental to interpreting the data. I'll provide screenshots soon.

@dmos62 I would see spikes in burned BSQ not as outlier but as valid events, just like BTC price peak with 20k was not an outlier... To be able to see more details a time scale selector will help to move out the spike from the view and get back the details of the lower values.

Regarding library: Bisq has the policy to be very conservative/restrictive to add new dependencies because of security reasons. Any library could containt malicious or vulnerable code which might be exploited at an app which deals with money. You are probably aware of the Node.js hack a while back, so such risks are real. The library you mentioned is exactly the type of library we try to avoid (and even remove those we have already added) as its a small 1-2 dev library which lacks of security audit. Even if the library code is rather small and non-critical we likely inroduce more transient dependencies which we also want to avoid. If a library comes from projects or companies with a strong security background we can be more tolerant as many eyes are on such libs. I understand the frustration to re-implement features but the potential financial risks are not justified by improved UI features. Hope you find a way to implement the intended features without requiring a new lib.

@chimp1984 thanks for taking the time to explain Bisq's new dependency policy. I appreciate the project's commitment to security.

Regarding outliers in burned BSQ, your input is very useful and welcome. So far I've presumed that the rarest outliers (<1% at the moment) are of little consequence analytically, and thus can be put out of view; but, it's not necessarily so. Do you have some use cases for charting outliers that you could share? Actually, if you could share any type of analysis that you would like to do using these charts it would be helpful in me getting a better picture of the requirements.

The recent spikes have been caused by the Bisq donation address owner who buys BSQ with the BTC he received from trade fee and from trades ended up in arbitration and then burn the BSQ. So that is a short term event with rather high amounts and a totally valid case and very important to be displayed in the chart.

For me the most interesting of the issuance and burn BSQ charts is to see the ration of both. If we burn more BSQ as we issue we are break-even/profitable. The issuance has cycle of about 1 month, the burning is a continuous stream (trade fees) with the above mentioned spikes.

I have not thought more in depth of the details but I think a time period slider where one can select any time range interested in would be very useful. If that is too much effort a simple time selector as we have in the trade statistics charts would be ok as well.

This is a long one; skip to conclusion for a summary.

I've implemented automatic re-ranging of the Y axis to the inliers (preventing chart distortion due to outliers). Screenshot [0]. The implementation took much longer than expected, largely because I'm new to the codebase and I'm new to Java development.

[0] BSQ burned chart ranged to inliers, notice the 3 outliers that are outside the chart's Y axis.

Also, during this time I've had more feedback, as well as time to reflect. Below I wrote down my current perception of the problem, including, in the conclusion section, what I see as the next steps.

Outliers are significant

While finishing up this feature, it became apparent, as pointed out by @chimp1984. that what I've labeled as outliers are significant as well. While, these outliers are on a different scale than the rest of the data points, thus difficult to compare, they are still interesting to compare among themselves. They are "valid events" as @chimp1984 put it.

Since this feature "pushes" outliers off-screen, as seen in the screenshot [0], it makes comparing the magnitudes of outliers impossible. Only their frequency can be comfortably assessed. A tooltip would allow to see the exact values, but the objective is to be able to make visual comparisons.

There is an immediate solution, which is to implement a UI switch, to switch between inlier-only mode (what I've implemented so far) and the normal all-data mode. That would effectively be switching between being able to examine the inliers and being able to examine the outliers. It could be a checkbox labeled "Set range to inliers".

Another look at the use cases

Currently, based on feedback from @ripcurlx and @chimp1984, I assume that there are 2 major use cases for these charts:

1) compare the rate of BSQ issue to the rate of BSQ burn, because it's an indicator of the project's ability to sustain its contributors (correct me if this could be phrased better); and, 2) independently examine the trends in BSQ issued and BSQ burnt, because BSQ issue and burn are distinct processes, and thus their understanding requires independent analysis.

Another look at the current state of charts

Currently, using charts (and not figures), we can examine the outliers in the BSQ burnt chart [1] (because the inliers are distorted to flatness), and we can examine the trends in BSQ issue [2]. That's the realm of use case 2). As far as 1) is concerned, rate comparison of BSQ issue and burn can't be effectively compared in the charts at this time.

[1] Current (two week old to be more accurate) BSQ burnt chart: only outliers can be examined. BSQ burned chart

[2] Current BSQ issued chart: apart from 0 not being on the Y axis, good chart; note that 1 month interval is used, because BSQ is issued in approximately 1 month intervals.

In this context, the feature I've implemented, automatic re-ranging to inliers [0], allows examining the BSQ burnt inliers, but not outliers. So on it's own it's not enough to solve use case 2).

BSQ burnt noise and log scale

Looking at the new screenshot [0], these problems can be seen:

the daily BSQ burnt series is logarithmicly distributed, but linear scale is used (because JavaFX doesn't support logarithimic axis);
the daily BSQ burnt series is noisy, which makes trend analysis difficult;
as mentioned, outlier magnitudes are unknown.

Implementing logarithmic charts without new dependencies is a big project, so we're stuck with linear at least for a while.

There might be an elegant solution to fixing the other two problems, however. A moving average would help with the noise, and, maybe more importantly, would visualise the influence of the outliers (which could be considered as extreme noise) on the rest of the trend.

Time-frames and intervals

Observations pertaining to time-frames and intervals:

BSQ is issued once a month, approximately, so finer intervals aren't interesting for this series;
BSQ is burnt often, and daily interval seems like the fastest interesting interval;
To compare BSQ issued and burnt, both series should be in the same (monthly) interval;
Since we're displaying all the data we have, with time it will get more and more horizontally squeezed;
Current effective time-frame is around 9 months;
On the daily BSQ burnt chart the points are already a bit squeezed, but being able to switch to a 6 or 3 month time-frame, in my opinion, would not give a substantial increase in analytical capability (it will later, however).

Citing the last observation, I think implementing different time-frames is less needed at this time than some other changes.

Also, since the independent analysis of BSQ issue and burn and the comparison of BSQ issue and burn have different time interval requirements, we probably need a third chart, dedicated to comparing BSQ issue and burn in the monthly interval. That would be the start of accomodating use case 1).

Conclusions and next steps

We discussed the following use-cases: 1) compare BSQ issue and burn, and 2) do independent analysis on both BSQ issue and burn. To accomodate them, the charts need a lot of reworking: currently the charts do not allow comparison of BSQ issue and burn, and the analysis of BSQ burnt is also severely lacking.

The automatic re-range to inliers feature I'm introducing now is only one of the steps towards effective charting of the BSQ burnt series. If it were merged, what would be lacking is outlier magnitude information, and a moving average or something similar to gauge the influence of outliers to the trend and for dealing with noise in general.

Proposals:

to fix BSQ burnt chart, a UI switch to switch between inlier range and full-range, so as to be able to examine both inliers and outliers, even if not at the same time;
to fix BSQ burnt chart, a moving average, so as to extract trend information from the noisy series;
to accomodate for use-case 1), I propose a third chart with two lines visualizing monthly BSQ burn and issue, one on top of the other;
implement time-frame/interval controls after that.

Note: I plan to now implement the UI switch before submitting a PR. Without the UI switch, the ability to examine outliers' magnitudes is lost, so it would feel like 1 step forward, 1 step back.

Here's a GIF recording of the UI switch:

BSQ burned slider animation

Also, I didn't mention that the inlier filter has settings, the main one being distance from mean in standard deviations beyond which points are considered outliers:

10 standard deviations BSQ burned 10 stddevs inlier filter

5 standard deviations BSQ burned 5 stddevs inlier filter

In both cases a 5% trimmed mean and standard deviation were used. Trimming/truncation means to take away a percentage of max and min elements. A trimmed mean or standard deviation is more resilient to outliers.

Here's a GIF recording of the UI switch:

BSQ burned slider animation

Also, I didn't mention that the inlier filter has settings, the main one being distance from mean in standard deviations beyond which points are considered outliers:

10 standard deviations BSQ burned 10 stddevs inlier filter

5 standard deviations BSQ burned 5 stddevs inlier filter

In both cases a 5% trimmed standard mean was used. Trimming/truncation means to take away a percentage of max and min elements. A trimmed standard deviation is more resilient to outliers.

to fix BSQ burnt chart, a UI switch to switch between inlier range and full-range, so as to be able to examine both inliers and outliers, even if not at the same time;

:+1: but maybe we find a better naming for the toggle. Default should be all visible IMO.

to fix BSQ burnt chart, a moving average, so as to extract trend information from the noisy series;

:+1:

to accomodate for use-case 1), I propose a third chart with two lines visualizing monthly BSQ burn and issue, one on top of the other;

If we want to communicate the relationship between issued and burnt BSQ I'm not sure if a stacked chart is the best choice. Maybe we could display a chart showing the relation of BSQ burnt : BSQ issued in percent?

implement time-frame/interval controls after that.

:+1:

@ripcurlx For the BSQ burnt v. BSQ issued chart, I could have been clearer. I'm thinking of a simple multi-line chart, not a stacked area chart. Something like this:

multi-line chart example

In this case, I think two lines criss-crossing are preferable to one line that represents their ratio. It's not intuitive at first, but deriving a ratio drops some information and it has a noticable effect when reading the chart. For example, when the ratio gets closer to zero, is that because the burn increased, because the issue decreased, or they both did something simultaniously? Or, if the ratio doesn't move, is that because issue and burn are changing at the same rate, negating each others' flux, or because they're stationary? A two-line chart would provide this information in a simple way.

bisq-network / bisq