Implement a 'get_historic_feed_price' call

[startailcoon]

The UI would like to have the possibility to view historic feed price.

If not mistaken, haven't tested though, es_objects does store feed price.

Relevant code bits are

• https://github.com/bitshares/bitshares-core/blob/cdc2db30c2f06aaddbfda965ee270b99dc24e0aa/libraries/plugins/es_objects/es_objects.cpp#L307
• https://github.com/bitshares/bitshares-core/blob/cdc2db30c2f06aaddbfda965ee270b99dc24e0aa/libraries/plugins/es_objects/es_objects.cpp#L256-L257

Related UI Issue https://github.com/bitshares/bitshares-ui/issues/502

[Zapata]

There is indeed the feed information in ES. Below a sample entry:

{
  "_index": "objects-bitasset",
  "_type": "data",
  "_id": "2.4.3",
  "_version": 1944131,
  "_score": null,
  "_source": {
    "id": "2.4.3",
    "asset_id": "1.3.103",
    "feeds": "[[\"1.2.167\",[\"2019-08-03T13:52:18\",{\"settlement_price\":{\"base\":{\"amount\":157000,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":37728643,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":111550,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":4616051,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.376\",[\"2019-08-03T14:00:24\",{\"settlement_price\":{\"base\":{\"amount\":5125,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":1243517,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":111550,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":4616051,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.564\",[\"2019-08-03T14:07:06\",{\"settlement_price\":{\"base\":{\"amount\":358000,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":86279939,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":111550,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":4616051,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.2850\",[\"2019-08-03T14:13:06\",{\"settlement_price\":{\"base\":{\"amount\":1,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":240,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":\"5212770297\",\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":\"215709668686\",\"asset_id\":\"1.3.0\"}}}]],[\"1.2.14035\",[\"2019-08-03T13:26:51\",{\"settlement_price\":{\"base\":{\"amount\":10,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":2409,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":4,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":803,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.25010\",[\"2019-08-03T08:37:24\",{\"settlement_price\":{\"base\":{\"amount\":37,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":8871,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":74,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":14785,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.30889\",[\"2019-08-03T13:36:12\",{\"settlement_price\":{\"base\":{\"amount\":66,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":15971,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":396,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":79855,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.31359\",[\"2019-08-03T14:00:21\",{\"settlement_price\":{\"base\":{\"amount\":15840,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":3849473,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":111550,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":4616051,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.35248\",[\"2019-08-03T14:15:06\",{\"settlement_price\":{\"base\":{\"amount\":332,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":79949,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":\"5212770297\",\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":\"215709668686\",\"asset_id\":\"1.3.0\"}}}]],[\"1.2.91787\",[\"2019-08-03T14:03:45\",{\"settlement_price\":{\"base\":{\"amount\":413829,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":100000000,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":413829,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":80000000,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.130210\",[\"2019-08-03T14:14:03\",{\"settlement_price\":{\"base\":{\"amount\":346000,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":83409311,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":111550,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":4616051,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.133999\",[\"2019-08-03T14:01:06\",{\"settlement_price\":{\"base\":{\"amount\":83250,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":19993649,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":111550,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":4616051,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.158783\",[\"2019-08-03T14:09:54\",{\"settlement_price\":{\"base\":{\"amount\":1633408839,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":\"394628297200\",\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":\"15220939318\",\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":\"629862005148\",\"asset_id\":\"1.3.0\"}}}]],[\"1.2.551261\",[\"2019-08-03T14:15:15\",{\"settlement_price\":{\"base\":{\"amount\":\"5100000000\",\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":\"1222747900000\",\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":\"223100000000\",\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":\"9232102000000\",\"asset_id\":\"1.3.0\"}}}]],[\"1.2.661381\",[\"2019-08-03T14:05:57\",{\"settlement_price\":{\"base\":{\"amount\":276000,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":66990289,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":111550,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":4616051,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.711128\",[\"2019-08-02T23:30:12\",{\"settlement_price\":{\"base\":{\"amount\":423,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":100000,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":44415,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":10000000,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.768841\",[\"2019-08-03T14:06:03\",{\"settlement_price\":{\"base\":{\"amount\":16080,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":3892889,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":111550,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":4616051,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.814541\",[\"2019-08-03T13:30:21\",{\"settlement_price\":{\"base\":{\"amount\":399000,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":95182433,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":111550,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":4616051,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.987999\",[\"2019-08-03T14:15:30\",{\"settlement_price\":{\"base\":{\"amount\":165000,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":39562817,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":111550,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":4616051,\"asset_id\":\"1.3.0\"}}}]],[\"1.2.1031601\",[\"2019-08-03T14:13:42\",{\"settlement_price\":{\"base\":{\"amount\":415,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":100000,\"asset_id\":\"1.3.0\"}},\"maintenance_collateral_ratio\":1750,\"maximum_short_squeeze_ratio\":1100,\"core_exchange_rate\":{\"base\":{\"amount\":2417,\"asset_id\":\"1.3.103\"},\"quote\":{\"amount\":100000,\"asset_id\":\"1.3.0\"}}}]]]",
    "current_feed": {
      "settlement_price": {
        "base": {
          "amount": 10,
          "asset_id": "1.3.103"
        },
        "quote": {
          "amount": 2409,
          "asset_id": "1.3.0"
        }
      },
      "maintenance_collateral_ratio": 1750,
      "maximum_short_squeeze_ratio": 1100,
      "core_exchange_rate": {
        "base": {
          "amount": 111550,
          "asset_id": "1.3.103"
        },
        "quote": {
          "amount": 4616051,
          "asset_id": "1.3.0"
        }
      }
    },
    "current_feed_publication_time": "2019-08-02T23:30:12",
    "current_maintenance_collateralization": {
      "base": {
        "amount": 16863,
        "asset_id": "1.3.0"
      },
      "quote": {
        "amount": 40,
        "asset_id": "1.3.103"
      }
    },
    "options": {
      "feed_lifetime_sec": 86400,
      "minimum_feeds": 7,
      "force_settlement_delay_sec": 86400,
      "force_settlement_offset_percent": 0,
      "maximum_force_settlement_volume": 2000,
      "short_backing_asset": "1.3.0",
      "extensions": []
    },
    "force_settled_volume": 0,
    "is_prediction_market": false,
    "settlement_price": {
      "base": {
        "amount": 5212770297,
        "asset_id": "1.3.103"
      },
      "quote": {
        "amount": 215709668686,
        "asset_id": "1.3.0"
      }
    },
    "settlement_fund": 168951231386,
    "asset_cer_updated": false,
    "feed_cer_updated": false,
    "object_id": "2.4.3",
    "block_time": "2019-08-03T14:15:33",
    "block_number": 39757314
  },
  "fields": {
    "block_time": [
      "2019-08-03T14:15:33.000Z"
    ],
    "current_feed_publication_time": [
      "2019-08-02T23:30:12.000Z"
    ]
  },
  "sort": [
    1564841733000
  ]
}

I suggest to build a /feed_price_history endpoint that returns a formatted result (instead of the raw ES content). Those could be the swagger specs:

  "/feed_price_history":
    get:
      description: Get the computed medan price feed history and optionally the corresponding source price feeds for a specific asset on a date range.
      operationId: api.explorer.get_feed_price_history
      parameters:
        - in: query
          name: asset_id
          type: string
          required: true
          default: 'USD'
          description: Asset id to get feed price from
        - in: query
          name: from_date
          default: "now-1d"
          type: string
          required: false
          description: Start date range
        - in: query
          name: to_date
          default: "now"
          type: string
          required: false
          description: End date range
        - in: query
          name: with_details
          default: false
          type: boolean
          required: false
          description: Add the detail of the source feeds used for the median computation
      responses:
        '200':
          description: Array of feed prices sorted by feed publication time
        '500': 
          description: Error processing parameters
      tags:
        - api
        - feed_price

Exact return type still to be designed.

Does this match the need?

[Zapata]

I can work on this, please assign it to me.

[Zapata]

The currently used ES only stores the last entry of the feed. Not the history (es-objects-keep-only-current = true). So we can't get the old pricefeed history. Either we setup a new instance of ES, or we need to take the same approach as Zapata/bitshares-pricefeed-tracker and load the information from the operation history, then recompute the median prices.

I'll wait some feedback first before starting any development.

[sschiessl-bcp]

Do you have an estimate of storage requirements if the feed history is included additionally? Could be a lot due to frequency.

Short term: recompute. If storage is acceptable, switch to es-object internally. External API should be the same. Your defintion sounds good.

[startailcoon]

Just so we're on the same page, when you say then recompute the median prices., we are not able to get individual price feeds per feeder, but rather the median from all feeds?

The UI issue was, to my understanding, to visualize the feed price by each feeder. But I do see that it could pose an issue to store all that.

Storing a median would be acceptable in my opinion.

[Zapata]

If we go in the 'recompute the median prices' route, we will use the account history to retrieved the published feeds (we don't store anything more than what we already have). We retrieve all the individual feeds per feeder for an asset over a period, and we recompute all the medians for each price feed changes. This computed value is returned by the proposed API endpoint (median + price of all feeders). Nothing new is stored in the database. We may only cache in memory (in the explorer api side) the result to reduce computation and query load.

The drawback of this solution are:

• we may compute medians different that the one effectively computed by the core node (the algorithm may be slightly different than the core one);
• we will have to cache the computation as the data retrieval + computation takes a couple of seconds.

If we go in the "new ES instance that stores all the objects-bitasset values (not only the last)" the caller may choose using the with_details parameter if he wants or not the prices of all the feeder additionally to the median. Currently we have no ES instance setup correctly to store this data and the volume is unknown. If the option is activated we will get all the medians + the price feeds of every feeder in the ES index.

[sschiessl-bcp]

At the end, we (the UI) would like to have an API that provides this information. Please proceed with whatever way seems most reasonable to you.

Since you already have the recompute code, we might just go with that?