Resolve global and regional volume discrepancies at Coin Dance

[cbeams]

We know that Coin Dance's numbers, e.g. https://coin.dance/volume/bisq/ALL are incorrect when checked against our own Global trading volume numbers. @ripcurlx has reached out to Coin Dance, and I believe they've recently gotten back. This issue is about tracking the progress and doing whatever is necessary to get this reconciled.

[ripcurlx]

We just received following response from Coin Dance today:

Our time frames start on the Sunday and end on the Saturday. Eg. October 15 -> October 21 == "Week of October 21".

We do still see that there is a difference between our numbers and the ones you've provided in the above table.

Our process is fairly simple: Retrieve trades from https://market.bisq.io/api/trades?market=all&sort=asc&timestamp_from=XXX&limit=10 Continue retrieving until we have caught up (until we're seeing 0 results), using the oldest time stamp from the previous retrieval set. Summarize into weekly values for display purposes. We do permanently store all transactions. After doing a quick review, we did see that we're missing transactions from July and most of August (1498903668 to 1503517726). Looking further into this, we see that the returned results aren't changing between the following queries like we would expect: https://market.bisq.io/api/trades?market=all&sort=asc&timestamp_from=1498903668&limit=10 https://market.bisq.io/api/trades?market=all&sort=asc&timestamp_from=1503887527&limit=10 So it looks like the issue is that we're missing transaction previous to the TIMESTAMP_FROM that we're using because our limit is currently set to 100 in our system (10 in the example above). Increasing this limit shows us more in the opposite direction than we're expecting. We have been expecting results to START at the TIMESTAMP_FROM time stamp and carry forward until it hits the limit. In actuality, we're seeing the latest transaction and the list then works backwards until it hits ours limit, so we're missing transactions.

The best solution would be to update the trades API to return trades that start from TIMESTAMP_FROM and go forward until it hits the limit. This way, we won't miss any trades.

It is possible for us to update things on our side to make it work with the API's behaviour as it is now, but that would be less ideal and still likely be unexpected behaviour for anyone else using the trades API.

Let us know what you think.

[cbeams]

Paging @dan-da. Is this something you could address and possibly accommodate in https://github.com/dan-da/bitsquare_market?

[cbeams]

@ripcurlx, thanks for following up there. Could you reply to the Coin Dance guys and invite them to join us here on this GitHub issue? This seems pretty simple, I'm sure we can sort it out.

[ripcurlx]

@cbeams I've just invited them to join. Yeah, it would be great to have the same numbers soon everywhere 😄

[Klakurka]

Just let us know if you need any other information.

It shouldn't take more than a few hours to update the data on our side once the api has been updated.

[cbeams]

Thanks, @Klakurka! We'll keep you posted here.

[dan-da]

https://market.bisq.io/api/trades?market=all&sort=asc&timestamp_from=1498903668&limit=10 https://market.bisq.io/api/trades?market=all&sort=asc&timestamp_from=1503887527&limit=10

investigating.

[dan-da]

I fixed a couple of related problems. Both were introduced during the transition to using multiple basecurrencies. The two links above give different results now, as expected. I believe this issue to be resolved.

Please try, and let me know if any issues.

See:

• https://github.com/dan-da/bitsquare_market/issues/8
• https://github.com/dan-da/bitsquare_market/issues/9

[Klakurka]

Looks good to me.

We'll have someone update the data and give you a heads up once that's been done (should be within the next 24 hours or so).

[Klakurka]

We're expecting to be able to retrieve the first 100 transactions made on the platform using the following query yet we are not (we instead get the lastest 100 or something similar to that):

https://markets.bisq.network/api/trades?market=all&sort=asc&timestamp_from=1&limit=100

Is this an issue with the API or do we need to make a change to our query?

Thanks

[dan-da]

It was an api issue, again related to sorting but has been there a long time. thanks for the report. please try again. btw, you can also just omit the timestamp_from param.

[cbeams]

Thanks, @dan-da.

@Klakurka, please also note that we've just updated the URL for the markets API from https://market.bisq.io => https://markets.bisq.network.

[Klakurka]

Alright have a look: https://coin.dance/volume/bisq

Let us know if the data still does not match what you're expecting.

Note that we still don't show 0-values for weeks with no volume, which is something that we plan on doing.

[ripcurlx]

Hi @Klakurka! Thanks for updating to match our new endpoint. I did a quick comparison of the the week of 11/11/2017 (11.69 BTC) on your site and what I receive from our API (13.53 BTC).

Here's the raw Json data I receive from the API (https://markets.bisq.network/api/volumes?basecurrency=BTC&interval=day): { "period_start": 1509840000, "volume": "1.00000000", "num_trades": 4 }, { "period_start": 1509926400, "volume": "3.37060000", "num_trades": 19 }, { "period_start": 1510012800, "volume": "2.42800000", "num_trades": 11 }, { "period_start": 1510099200, "volume": "2.85310000", "num_trades": 12 }, { "period_start": 1510185600, "volume": "0.68000000", "num_trades": 4 }, { "period_start": 1510272000, "volume": "2.74500000", "num_trades": 12 }, { "period_start": 1510358400, "volume": "0.45000000", "num_trades": 4 }

And here printed more pretty:

Date	Volume in BTC
05/11/2017	1.00 BTC
06/11/2017	3.37 BTC
07/11/2017	2.43 BTC
08/11/2017	2.85 BTC
09/11/2017	0.68 BTC
10/11/2017	2.75 BTC
11/11/2017	0.45 BTC

So it seems there is still a difference of 1.84 BTC. Do you receive different data with the query you are using?

[dan-da]

Note that /volumes API is always returning volume of the basecurrency. otoh, the /trades API returns amount and price of left side of each market (left side is basecurrency when trading against fiat, else other cryptocurrency. eg btc_usd, xmr_btc).

So if trying to sum btc volumes using /trades API for comparison with /volumes, one must use amount field when trading against fiat and price * amount when trading against other cryptocurrency.

Using this methodology, I just verified that these two queries from the two APIs sum to the same amount:

https://markets.bisq.network/api/volumes?basecurrency=BTC&interval=day&timestamp_from=1510531200&timestamp_to=1510612509

https://markets.bisq.network/api/trades?market=all&timestamp_from=1510531200&timestamp_to=1510612509

Of course, when multiplying, there is a loss of precision. for summarization purposes either the /hloc or /volumes apis should be used rather than the /trades API.

[ripcurlx]

@dan-da: Thanks for clarification! @Klakurka: David, would this be doable from your side to use the /hloc or /volumes api for your visualizations?

[Klakurka]

We're actually not sure how to leverage the /volumes API for this as it only returns the total sum for the single currency entered (eg. BTC) so we're unable to get the breakdown by currency (USD, EUR, CAD, XMR, BCH, etc).

We picked out a random set of transactions for you to review to see if we're missing anything:

WHERE TIME STAMP IS BETWEEN 1509875308 and 1511038495 INCLUSIVE (124 trades):

` Time Stamp | Volume (fiat/alt) | Volume (BTC) | Fiat/Alt Currency Code

1509875308 | 6516.3 | 0.5 | EUR

1509881423 | 6459.22 | 0.3 | EUR

1509893719 | 7633.8 | 0.1 | USD

1509913183 | 6694.85 | 0.1 | EUR

1509929752 | 6969.82 | 0.05 | EUR

1509930876 | 0.01 | 19.63763 | XMR

1509935447 | 9583.87 | 0.026 | CAD

1509957411 | 7384.54 | 0.1315 | USD

1509961418 | 6262.36 | 0.5 | EUR

1509964574 | 6375.18 | 0.15 | EUR

1509967997 | 0.01 | 40.40829 | XMR

1509978888 | 6386.93 | 0.2 | EUR

1509980484 | 7300.03 | 0.1 | USD

1509981996 | 5692.17 | 0.02 | GBP

1509984741 | 7236.71 | 0.1 | USD

1509995907 | 6921.74 | 0.1 | USD

1509997304 | 7017.1 | 0.1 | USD

1509997354 | 7018.05 | 0.25 | USD

1509997415 | 7018.05 | 0.1 | USD

1509998796 | 7123.79 | 0.1 | USD

1510000394 | 7043.21 | 0.17 | USD

1510001995 | 6273.15 | 0.025 | EUR

1510011031 | 0.01 | 60.89771 | LTC

1510052914 | 5554.52 | 0.028 | GBP

1510053010 | 5555.85 | 0.1 | GBP

1510065328 | 6247.18 | 0.25 | EUR

1510065790 | 6249.13 | 0.25 | EUR

1510069658 | 0.04 | 9.382667 | ETH

1510080149 | 7103.47 | 0.2 | USD

1510081484 | 7111.35 | 0.2 | USD

1510081662 | 5505.53 | 0.1 | GBP

1510085205 | 6858.5 | 0.5 | USD

1510090827 | 7066.29 | 0.2 | USD

1510091613 | 6304.27 | 0.2 | EUR

1510110593 | 7300 | 0.047 | USD

1510123105 | 7409.02 | 0.33 | USD

1510144021 | 6424.61 | 0.1561 | EUR

1510146254 | 5625 | 0.02 | GBP

1510152214 | 7513.48 | 0.05 | USD

1510152307 | 6403.25 | 0.5 | EUR

1510153730 | 5815.14 | 0.2 | GBP

1510154545 | 6509.8 | 0.4 | EUR

1510174900 | 7248.59 | 0.25 | USD

1510175329 | 6520 | 0.3 | EUR

1510176603 | 6686 | 0.5 | EUR

1510184559 | 5697.8 | 0.1 | GBP

1510205638 | 0.02 | 1.89601 | XMR

1510229992 | 7086.78 | 0.2 | USD

1510246148 | 7181.65 | 0.25 | USD

1510263153 | 6490.32 | 0.2 | EUR

1510277231 | 0.05 | 2.125022 | DASH

1510298961 | 7174.65 | 0.25 | USD

1510301816 | 6269.43 | 0.1 | EUR

1510313380 | 6213.73 | 0.5 | EUR

1510314769 | 6245.12 | 0.5 | EUR

1510315255 | 0.05 | 1.934365 | DASH

1510332053 | 0.04 | 2.567394 | XMR

1510340287 | 6507.14 | 0.17 | USD

1510340841 | 8061.6 | 0.025 | CAD

1510340863 | 6474.96 | 0.5 | USD

1510349675 | 5656.82 | 0.15 | EUR

1510350180 | 6042.38 | 0.25 | EUR

1510417281 | 6138.42 | 0.05 | EUR

1510418889 | 5534.16 | 0.2 | EUR

1510439762 | 5699.14 | 0.1 | EUR

1510442912 | 5394.59 | 0.1 | EUR

1510473086 | 5169.75 | 0.14 | EUR

1510477462 | 5341.57 | 0.25 | EUR

1510481079 | 5550 | 0.18 | EUR

1510483923 | 5428.16 | 0.25 | EUR

1510486227 | 6248.3 | 0.05 | EUR

1510490919 | 6398.37 | 0.2 | CHF

1510496612 | 0.02 | 9.969175 | XMR

1510525576 | 4966.83 | 0.5 | EUR

1510536723 | 5473.28 | 0.25 | EUR

1510579542 | 6684.12 | 0.1 | USD

1510589863 | 0.02 | 15.37244 | XMR

1510592196 | 4991.55 | 0.2 | GBP

1510613671 | 5657.34 | 0.5 | EUR

1510624911 | 6550.33 | 0.05 | EUR

1510635858 | 6603.83 | 0.24 | USD

1510657724 | 5069.91 | 0.5 | GBP

1510683597 | 6832.44 | 0.5 | CHF

1510685564 | 5919.67 | 0.1 | EUR

1510696865 | 103499 | 0.0001 | ZAR

1510698276 | 103500 | 0.0001 | ZAR

1510700121 | 0 | 0.047619 | NBT

1510702334 | 103499 | 0.0001 | ZAR

1510708747 | 103799 | 0.0001 | ZAR

1510709029 | 0 | 0.05 | NBT

1510710007 | 0 | 0.05 | NBT

1510719326 | 0.02 | 26.8099 | XMR

1510725759 | 0.05 | 4.867324 | ETH

1510731259 | 9486.38 | 0.0001 | AUD

1510733937 | 5856.01 | 0.5 | EUR

1510741446 | 5783.16 | 0.142 | EUR

1510743414 | 5716.61 | 0.14 | EUR

1510745889 | 7084.78 | 0.25 | USD

1510746087 | 5900 | 0.5 | EUR

1510751916 | 0.02 | 27.79549 | XMR

1510783289 | 5986.54 | 0.5 | EUR

1510787040 | 7283.08 | 0.5 | USD

1510803526 | 7185.48 | 0.0001 | USD

1510803888 | 157905 | 0.0001 | CZK

1510815188 | 7261.88 | 0.2 | USD

1510818378 | 5969.77 | 0.166 | EUR

1510824118 | 6600 | 0.1 | EUR

1510839399 | 7398.08 | 0.5 | USD

1510847032 | 6389.62 | 0.5 | EUR

1510850620 | 10007.1 | 0.0001 | AUD

1510870231 | 6458.94 | 0.25 | EUR

1510876783 | 0.02 | 31.5009 | XMR

1510889900 | 8091.52 | 0.05 | USD

1510895288 | 7943.05 | 0.25 | USD

1510932599 | 7907.06 | 0.25 | USD

1510950411 | 7923.63 | 0.05 | USD

1510960965 | 6521.85 | 0.076 | EUR

1510975237 | 7647.66 | 0.002 | USD

1510978394 | 7707.25 | 0.0014 | USD

1511005010 | 0.01 | 21.96004 | LTC

1511018376 | 6557.46 | 0.1 | EUR

1511021133 | 5997.54 | 0.03 | GBP

1511030179 | 7776.85 | 0.25 | USD

1511038495 | 6583.33 | 0.25 | EUR `

It seems to match up with https://markets.bisq.network/api/volumes?basecurrency=BTC&interval=day&timestamp_from=1509875308&timestamp_to=1511038495. Can you confirm?

[ripcurlx]

At a first glance it looks like you have mixed up the columns for Alt Coins. e.g.

Time Stamp	Volume (fiat/alt)	Volume (BTC)	Fiat/Alt Currency Code
1509929752	6969.82	0.05	EUR
1509930876	0.01	19.63763	XMR

Shouldn't this be?

Time Stamp	Volume (fiat/alt)	Volume (BTC)	Fiat/Alt Currency Code
1509929752	6969.82	0.05	EUR
1509930876	19.63763	0.01	XMR

Still, putting it in the right order and summing it up on a daily basis I'll get:

Date	Coin.Dance	Volume from API
5/11	1	1
6/11	2.1525	3.3706
7/11	2.068	2.428
8/11	2.8531	2.8531
9/11	0.67	0.68
10/11	2.585	2.745
11/11	0.45	0.45
12/11	1.59	1.77
13/11	1.07	1.35
14/11	1.3903	1.3904
15/11	2.6222	3.7832
16/11	1.7363	2.2164
17/11	0.676	0.676
18/11	0.6434	0.5834

It seems to be that there is always a difference in numbers if there were also Altcoin trades on this specific day.

To get the volumes per currency (e.g. USD) you should be able to add the market to the query: https://markets.bisq.network/api/volumes?basecurrency=BTC&market=btc_usd&interval=day&timestamp_from=1509875308&timestamp_to=1511038495

Unfortunately it seems that this parameter is not working as I would expect from the API documentation at the moment: https://markets.bisq.network/api/#volumes

@dan-da:Could it be that the market parameter is ignored for the volumes endpoint?

[dan-da]

Could it be that the market parameter is ignored for the volumes endpoint?

yes and no. The docs are a bit misleading. At present, the market param is only used to identify the basecurrency, but then volumes are returned for the entire basecurrency as if it were passed ?basecurrency flag. market flag was added as a convenience because this API is the only one to use basecurrency concept, all others use market. Filtering result data by market was not contemplated when the API was being implemented, though in hindsight it seems to make sense. It would take a little effort to implement though.

-- edit -- Filtering /volumes results by market would be functionally very similar to /hloc. The difference is that /hloc returns additional data, and also the base currency could be on left or right of market pair depending on if trading against fiat or not. ie,

base_currency_volume = type(other) == fiat ? volume_left : volume_right.  

The type data can be found in the /markets api. (ltype, rtype) -- end edit --

A couple other thoughts for this use case:

1. maybe the /hloc API could be used? It provides a time window and volumes for both sides of market. The only thing is that each market must be queried separately.

2. The /trades API could easily be extended to include volume. That is likely the easiest/best solution. The reason it doesn't is because I looked at what other exchanges were doing at the time (btc-e, bitfinex, etc) and they included price, amount but not volume so I followed suit. Also, if returning thousands of trades, each extra field contributes to size of result set and we want to minimize it.

Before making any api change though... I am curious how coin.dance handles this for other exchanges that do not include volume in trades api.

Example:

https://docs.bitfinex.com/v1/reference#rest-public-trades

[{
  "timestamp":1444266681,
  "tid":11988919,
  "price":"244.8",
  "amount":"0.03297384",
  "exchange":"bitfinex",
  "type":"sell"
}]

I will be travelling in a few hours. I could add volume field within 2-3 days if there is consensus that is best solution.

[ripcurlx]

@dan-da: Thanks for clarification 👍 @Klakurka: Hi David!

Before making any api change though... I am curious how coin.dance handles this for other exchanges that do not include volume in trades api. Do you have this case for other exchanges you are monitoring or are you accessing their data using different endpoints? If we would go for @dan-da's suggestion adding volume to the /trades API, I guess this would be your preferred solution, wouldn't it?

[Klakurka]

Each service has provided us with a trades API in more or less the same way. The only difference with the Bisq API is that we have to parse and separate out the exchanging currency from the 'market' property. This is fine for us.

I don't fully understand why there is still a discrepancy beyond the incorrect ordering of the currency columns in a few cases though, or what adding 'volume' to the trades API would buy us. It's expected that we should be able to retrieve all volume data by summing up the trades completed.

Could you clarify?

[dan-da]

From above comment:

So if trying to sum btc volumes using /trades API for comparison with /volumes, one must use amount field when trading against fiat and price * amount when trading against other cryptocurrency.

Of course, when multiplying, there is a loss of precision.

adding volume field would avoid the multiplication and resulting loss of precision. Eg The difference between 6 btc and 5.99999999

[ripcurlx]

@Klakurka: Sorry for my late reply, but I was sick the last couple of days. I should have done it way earlier, but finally I did the calculations from the trade API myself for the mentioned days above and besides very small rounding issue show the same numbers like the volume API. Please see the testing sheet (https://docs.google.com/spreadsheets/d/1o-I5fAx7DJRVqYjW8fPbo0ztlGIhIZ1EM2iLc5aEHnA/edit#gid=2001119938) for how I did it exactly.

Date	Coin.Dance	Volume from API	calc. Volume from API
8/11	2.8531	2.8531	2.853
9/11	0.67	0.68	0.68
10/11	2.585	2.745	2.745
11/11	0.45	0.45	0.45
12/11	1.59	1.77	1.77
13/11	1.07	1.35	1.35
14/11	1.3903	1.3904	1.3904
15/11	2.6222	3.7832	3.7832
16/11	1.7363	2.2164	2.2164
17/11	0.676	0.676	0.676

Could you please check if you calculate the btc volume for alt coins in a different way than using price * amount?

[ripcurlx]

@Klakurka: Hi David! Did you have time to have a look at the how I'm using the Trades API endpoint in my example sheets? If you do it like that it everything should be calculated fine (rounding errors are marginal).

[cbeams]

I'm not sure where we should go from here. It looks like numbers at https://coin.dance are still skewed.

@Klakurka, is there anything we can do to make it easier for you guys to get this fixed? We'd really love to be able to point people to Coin Dance, but we of course cannot do that (and are even discouraging people from doing so) until this is worked out.

Thanks!

[ripcurlx]

@cbeams: @Klakurka told me that they will look into the numbers soon again. They just didn't have the developer resources to do so.

[Klakurka]

We've got someone looking into this one now.

[Klakurka]

Have a look https://coin.dance/volume/bisq

[cbeams]

Thanks, @Klakurka! @ripcurlx should be able to confirm these numbers when he's next available.

[ripcurlx]

Hi @Klakurka! I've checked the week of 23rd of Dec (17th - 23rd). In the Weekly Bisq Volume (Global) on coin.dance I see a BTC value of 34.09 and USD 587,354. In our reporting sheet I see for the same week 65.92 BTC and 1,204,387,27 USD. So it seems the charts are still off. In the reporting chart linked in this comment you'll see the daily numbers of December. If I can help with anything else, please let me know!

Best,

Christoph

[Klakurka]

I took a look at our transaction data with an offset takes into account the time zone difference and see a total volume of 65.9188 BTC for that period (which looks good).

The issue is that we're only counting FIAT <-> BTC on Bisq in the summary ('all') chart so we'll get that fixed for you.

Can you confirm that the individual fiat currencies that we list (the blue-coloured charts) are as expected? We currently only show currencies with reasonable volume, although we still use every fiat currency for the total at the top (although, again, we need to change this to include BTC <-> alt as well).

[edit] Just keep in mind that we're going to be off by ~8 hours as our numbers are based on PST.

[Klakurka]

The global chart should now be working as expected. Note that we get 51.76 BTC instead of 65.92 BTC due to the 8 hour time zone difference, which we've confirmed is correct for the transactions that we have (which includes transactions between 1513499090976 and 1514065296699 inclusive).

https://coin.dance/volume/bisq/ALL

[ripcurlx]

@Klakurka Looking good now! Thanks for all your efforts David! I'm closing this issue and wish you and everyone at coin.dance a happy new year!