wrong values being sent?

[gsiisg]

I am trying to calculate the profit of crafting food items and see which is more profitable daily, so I need to update many items. I already use the suggested format https://west.albion-online-data.com/api/v2/stats/prices/{concat_material_names} and the update works great, it can return many item updates with 4096 characters.

However, I find that some items' prices are incorrectly returned.

I usually type the names of each of the items I want into the market UI, then scroll through pages of it to get the higher enhancement version prices. However, some of the prices returned are from items in the middle of the page instead of the top, even though I wrote code to grab "sell_price_min" from the json object. I'm wondering if this is known behavior sometimes, or unique to my situation since I upload a ton of prices at a time.

I even sometimes get the 904 error saying too much data...

[phendryx]

904's can happen from time to time but is usually only on history/gold. That can be corrected on the server side but there needs to be some limit because in the past, spammers would just upload a huge packet of incorrect data. 904's, based on logs, are fairly rare but do happen. But again, can be looked into.

What's sent to ingest sometimes has negative effects on the API. When we query data from the database, we bin it in 5 minute buckets and return only the most recent bucket. What can happen here is if all your data hasn't made it to the server yet or crosses over multiple 5 minute buckets, then the data before the most current bucket doesn't get included in the response. If we do not bin the data at all, then we return all data for the requested filters that has been seen in the last 24 hours, which isn't a bad thing, except, we do not know if orders were sold, removed, etc. So we might then also return data that no longer exists in the market. So either way, there's a problem. The decision to bin data in 5 minute buckets was made before my time maintaining the project and I'm open to suggestions but it'd have to be a larger conversation in discord with probably a poll or something. Maybe even a v3 of the API that supports both binned and non-binned data.

[gsiisg]

Do the clocks sync with UTC time for the 5 min bin? Can I avoid uploading data at the 5 min mark so no data gets truncated across the 5 min bin? I can't seem to control when the sending of data, is there a toggle for it? It's when I see the message "INFO[2024-05-28T21:55:32-07:00] Successfully sent ingest request to http://pow.west.albion-online-data.com", right? Maybe the bundles can be formed two at a time, if the same item was in the previous bundle then not include it again, since usually the 2nd bundle will have items later down the list etc. (Assuming we want the top of the list, maybe a toggle to change behavior to bottom of the list if someone is interested in those)

[phendryx]

The AODP server is UTC time, so yes, give or take a few milliseconds.

The client uploads after if processes the POW, which can be a random amount of time, but usually not minutes unless you just mash the "next" button on the market. No, you can not control it, it's done as data is received by the game. The data is not truncated, it's still in the database, it's just not included in the API results.

It would be overkill to rewrite the client side or server side of things to process "2 bundles at a time". Adjusting the API would make more sense. And again, that changes how the current results are generated which is how people expect data at the moment so there needs to be a larger discussion in discord about how that could work.