Optimized API for tools only interested in current price

[Kouzukii]

In order to alleviate universalis server congestion we could employ a more optimized API for tools that are simply interested in the current price rather than a history of recorded prices, which is what is currently provided by the "Market board current data" API.

An example request and response could looks like:

Request:	parameter	value	description
itemIds	e.g. 29685,33673	Items the client is interested in
worldDcRegion	e.g. 40 or chaos or europe	Homeworld/Datacenter/Region of the client
scope	multiple of: world, datacenter, region	In which scopes the client is interested in, e.g. price in homeworld, cheapest in datacenter, cheapest in region etc. Scopes are only valid if worldDcRegion contains info on it.
include	multiple of: recentPurchase, minListing, medianListing, averageSalePrice, dailySaleVelocity	What fields the client is interest in

Response:

{
  "results": [
    {
      "itemId": 1,
      "nq": {
        "minListing": {
          "world": {"price": 123},
          "dc": {"worldId": 40, "price": 123},
          "region": {"worldId": 80, "price": 123}
        },
        "medianListing": {
          "world": {"price": 123},
          "dc": {"price": 123},
          "region": {"price": 123}
        },
        "recentPurchase": {
          "world": {"price": 123, "timestamp": 12345679},
          "dc": {"worldId": 40, "price": 123, "timestamp": 12345679},
          "region": {"worldId": 80, "price": 123, "timestamp": 12345679}
        },
        "averageSalePrice": {
          "world": {"price": 123.45},
          "dc": {"price": 123.45},
          "region": {"price": 123.45}
        },
        "dailySaleVelocity": {
          "world": {"quantity": 1.23},
          "dc": {"quantity": 1.23},
          "region": {"quantity": 1.23}
        }
      },
      "hq": {/* same format as nq */},
      "worldUploadTimes": [
        {
          "worldId": 40,
          "timestamp": 123456789
        },
        {
          "worldId": 80,
          "timestamp": 123456789
        }
      ]
    },
    {
      "itemId": 2,
      ...
    }
  ],
  "failedItems": [3, 4, ...] // if item is not marketable or an unexpected error occurred
}

Fields and Scopes are only included when requested by the client.

[karashiiro]

Might be missing something - how does this differ from calling the existing current data endpoint with &entries=0?

[Kouzukii]

The current endpoint only includes information for a single world OR the entire datacenter OR the entire region. That would require 3 separate requests if the user would want to know the price of an item for both their homeworld and a potential server hop destination. Also setting entries=0 and listing=0 does not tell you which world has the cheapest price or most recent purchase.

[Kouzukii]

Which is why currently I make a request to the largest scope, e.g. their Datacenter and then find their homeworld within the response to display both the cheapest listing on their homeworld and the datacenter.

[karashiiro]

Makes sense, I can see the argument from an API consumer perspective; this makes it a lot more intuitive to do stats efficiently for clients - but if you're able to request a large scope and filter on the result already, how would this improve things for e.g. PriceInsight?

Also setting entries=0 and listings=0 causes averagePrice and saleVelocity to be 0

This sounds like a bug actually, statsWithin should take precedence for the DB fetch

[Kouzukii]

Also setting entries=0 and listings=0 causes averagePrice and saleVelocity to be 0

This was my mistake, strike that 😅

[karashiiro]

Wait it's working fine, yeah - it's just because I removed sales from the current data endpoint to debug load issues...

[Kouzukii]

The optimization aim here is not to improve PriceInsight, but rather optimize performance of the Universalis API. Since when only requesting these minimal datapoints you could use more optimized Database queries for aggregation, rather than aggregating the Data within the Universalis Application Server (at least that was the state when I made my last PR which was 2 years ago, things might be different now)

[karashiiro]

Oh, you'd push the aggregation down to the DB itself - that makes much more sense. Alright, that makes sense to me, no objections from an implementation standpoint either, then.

[Eisenhuth]

one thing that might also be worth considering could be adding median prices (hq, nq)

I've got a few requests that I could replace entirely with the above if all of those values were present

[Kouzukii]

Any ideas for a good name for the API? I'd suggest something like CurrentPrice or AggregatedMarketBoardData

[cohenaj194]

It might be a good idea to make a single api call that can showcase all item stats in a single call instead of requiring multiple calls.

I have something similar in saddlebag:

• swagger : https://app.swaggerhub.com/apis/SaddlebagExchange/ffxiv/1.0.0#/default/post_api_ffxivrawstats
• postman: https://www.postman.com/restless-zodiac-865092/workspace/saddlebag-exchange-public-api/request/21652432-e79495b9-f558-44a7-a33a-814037054274?tab=body

[karashiiro]

Any ideas for a good name for the API? I'd suggest something like CurrentPrice or AggregatedMarketBoardData

AggregatedMarketBoardData sounds good, CurrentPrice seems too similar to CurrentlyShown

[nan42]

I believe we can also use fields for this with some changes in how it's handled. It can be used to exclude listings and recentHistory arrays. It will not require an entirely new API and can still have an optimized query/caching internally.

[nan42]

Here's an example with fields=items.itemID,items.worldID,items.currentAveragePrice,items.regularSaleVelocity,items.stackSizeHistogram (request URL)

and the response:

{
  "items": {
    "7": {
      "itemID": 7,
      "worldID": 39,
      "currentAveragePrice": 70.81579,
      "regularSaleVelocity": 142.85715,
      "stackSizeHistogram": {
        "5": 1,
        "241": 1,
        "400": 1,
        "407": 1,
        "700": 2,
        "1000": 11,
        "2000": 6,
        "2040": 1,
        "2500": 1,
        "3550": 4,
        "4000": 1,
        "4400": 1,
        "4440": 1,
        "5000": 1,
        "5060": 1,
        "5950": 1,
        "5999": 1,
        "6240": 1,
        "9480": 1
      }
    },
    "8": {
      "itemID": 8,
      "worldID": 39,
      "currentAveragePrice": 153038.14,
      "regularSaleVelocity": 571.4286,
      "stackSizeHistogram": {
        "1": 7,
        "100": 4,
        "112": 1,
        "311": 1,
        "400": 1,
        "500": 3,
        "621": 1,
        "1000": 4,
        "1440": 1,
        "1500": 1,
        "2000": 7,
        "2500": 1,
        "2865": 1,
        "2880": 1,
        "3000": 4,
        "3018": 1,
        "3600": 1,
        "4000": 1,
        "4380": 1,
        "4669": 1,
        "5000": 1,
        "5210": 1,
        "9000": 3
      }
    }
  }
}

We can make this more client-friendly with macro expansion, e.g. instead of the above fields=... the client would requests fields=@stats and the API would automatically expand it into the proper fields depending on the context.