search

ao-data / albiondata-client

Distributed client for the Albion Online Data project.
https://www.albion-online-data.com/
MIT License
106 stars 32 forks source link

Wrong data in the history #118

Open sssuhanov opened 2 months ago

sssuhanov commented 2 months ago

Sometimes, the history API returns wrong numbers. I don't know if it is a problem on the client or server side, but data from API is different from data in the market.

Examples:

I checked data on different websites who is using the project, and they have the same issue: https://albionfreemarket.com/pricecheck/T8_2H_TOOL_AXE_AVALON

image
phendryx commented 2 months ago

Any downstream sites would have the same issue if the data displayed on those sites is coming from the AODP API.

I see "bad" data in the database. image

The problem is, I don't know how it gets there. The last time it occurred for this specific item in this location was Sept 1st, which we only hold logs for a few days. So, I agree with you, but without recent findings, say less than 24 hours old, it's hard to say what happened or how.

That being said, I'm going to leave this open and either increase log retention or come up with a clever way to detect it, monitor when it's detected and alert so I can look at it. The problem usually is, as is this time, we find it too late and there's no way to know exactly what happened.

sssuhanov commented 2 months ago

I collected data on my side and have agregated data. I can easily find recent examples for you.

Example: https://west.albion-online-data.com/api/v2/stats/charts/T4_HEAD_CLOTH_HELL@3.json?locations=Thetford&date=15-08-2024&qualities=4&time-scale=24

[ { "location": "Thetford", "item_id": "T4_HEAD_CLOTH_HELL@3", "quality": 4, "data": { "timestamps": [ "2024-08-14T00:00:00", "2024-08-15T00:00:00", "2024-08-16T00:00:00", "2024-08-17T00:00:00", "2024-08-18T00:00:00", "2024-08-19T00:00:00", "2024-08-20T00:00:00", "2024-08-21T00:00:00", "2024-08-22T00:00:00", "2024-08-23T00:00:00", "2024-08-24T00:00:00", "2024-08-25T00:00:00", "2024-08-26T00:00:00", "2024-08-27T00:00:00", "2024-08-28T00:00:00", "2024-08-29T00:00:00", "2024-08-30T00:00:00", "2024-08-31T00:00:00", "2024-09-01T00:00:00", "2024-09-02T00:00:00", "2024-09-03T00:00:00", "2024-09-04T00:00:00" ], "prices_avg": [ 97069, 94853, 97102, 96945, 98793, 94778, 96213, 97771, 103638, 98028, 111362, 109194, 103865, 99087, 403, 386, 390, 378, 380, 377, 386, 419 ], "item_count": [ 44, 50, 47, 44, 43, 55, 43, 73, 39, 60, 58, 43, 31, 42, 81830, 60733, 70174, 85884, 105255, 71868, 63127, 46263 ] } } ]

In the game: image

More examples:

phendryx commented 2 months ago

So, I found the incoming logs directly from the client that submitted that data. The data looks right, in that the server processed the data the way it should have. The client is apparently having issues, which is also hard to track down. For market history, we receive 2 packs of data; information about the item (id, location, etc, https://github.com/ao-data/albiondata-client/blob/master/client/operation_auction_get_item_average_stats.go#L19) and a packet of datetime/count/silver (https://github.com/ao-data/albiondata-client/blob/master/client/operation_auction_get_item_average_stats.go#L49). We do some weird stuff to keep track of those and I suspect the bug is in there. This isn't a new thing but I also haven't seen it brought up in a while. I'll dig into it this weekend and see if I can prove failure or come up with a better way to track it.