remove core reliance on get_state

Condenser uses steemd's get_state to fetch a large state tree, provided a URL path. Internally, get_state branches on request path and makes a number of API calls, bundling them into a single response. (view implementation)

Issues with this approach

core reliance on get_state contributes to condenser inflexibility
the data get_state provides is being moved to a number of different services
with appbase release, get_state is deprecated
extraneous data we don't use

General response format

The response is saved in condenser's state.global.

{
  "current_route": <url>,
  "props": <get_dynamic_global_properties()>              // basic chain properties. a few keys are relevant
  "tag_idx": {
    "trending": <get_trending_tags().map(name)>           // simple array of trending tags used on homepage etc
  },
  "tags": <get_trending_tags()>,                          // only used for tag "explorer" page
  "content": {*url: post_object}                          // posts keyed by author/permlink
  "accounts": {*name: account_object}                     // relevant accounts keyed by name
  "pow_queue": [],                                        // unused/ignore
  "witnesses": <get_witnesses_by_vote()>,                 // only used for /~witnesses
  "discussion_idx": {....},                               // ordered post keys, keyed by (1) current tag (or ""), (2), active sort (e.g. created/trending)
  "witness_schedule": {<get_witness_schedule_object()>}   // unused/ignore
  "feed_price": {<get_current_median_history_price()>}    // referenced from: wallet, market, and signup_bonus(unused?)
  "error": ""
}

Distinct request types

The structure changes depending on the request type. I will document these in greater detail later.

RT1. lists of posts (e.g. /created, /trending, /@user/feed, @user/blog) RT2. discussion thread (/category/user/permlink) RT3. tag "explorer" (/tags) RT4. witness page (/~witnesses) RT5. account history (/user/transfers) RT6. "invalid" / misc -- condenser calls get_state on every URL, even for paths which it does not explicitly handle (e.g. /market). in these cases, steemd returns a basic skeleton which does includes the props key and feed_price key, which condenser uses.

First course of action

Handle the RT6 "invalid" case, i.e. do not call get_state on pages it's not meant to explicitly handle. This means we need to remove core reliance (which is fairly minimal, as shown below) on state.global and move general info into tidy branches.

Here's the get_state response for invalid URLs:

  {
    "current_route": "market",
    "props": {
      "id": 0,
      "head_block_number": 18517158,
      "head_block_id": "011a8ca67310dc8d9b8dd9adf5191df15a13f82d",
      "time": "2017-12-29T18:18:21",
      "current_witness": "netuoso",
      "total_pow": 514415,
      "num_pow_witnesses": 172,
      "virtual_supply": "263378342.088 STEEM",
      "current_supply": "261923616.580 STEEM",
      "confidential_supply": "0.000 STEEM",
      "current_sbd_supply": "4556200.292 SBD",
      "confidential_sbd_supply": "0.000 SBD",
      "total_vesting_fund_steem": "196967804.024 STEEM",
      "total_vesting_shares": "403667516864.126525 VESTS",
      "total_reward_fund_steem": "0.000 STEEM",
      "total_reward_shares2": "0",
      "pending_rewarded_vesting_shares": "304803594.287417 VESTS",
      "pending_rewarded_vesting_steem": "148044.525 STEEM",
      "sbd_interest_rate": 0,
      "sbd_print_rate": 10000,
      "maximum_block_size": 65536,
      "current_aslot": 18578767,
      "recent_slots_filled": "340282366920938463463374607431768211455",
      "participation_count": 128,
      "last_irreversible_block_num": 18517138,
      "vote_power_reserve_rate": 10,
      "current_reserve_ratio": 145928382,
      "average_block_size": 15677,
      "max_virtual_bandwidth": "192801418845880320000"
    },
    "tag_idx": {
      "trending": [
        "",
        "life",
        "photography",
        "steemit",
        "kr",
        "art",
        "bitcoin",
        "introduceyourself",
        "blog",
        "travel",
        "steem",
        "spanish",
        "cryptocurrency",
        "food",
        "story",
        "nature",
        "news",
        "funny",
        "dtube",
        "indonesia",
        "writing",
        "busy",
        "utopian-io",
        "christmas",
        "money",
        "cn",
        "crypto",
        "sevendaybnwchallenge",
        "photo",
        "video",
        "contest",
        "love",
        "music",
        "aceh",
        "colorchallenge",
        "meme",
        "poetry",
        "health",
        "new",
        "tr",
        "blockchain",
        "science",
        "dmania",
        "technology",
        "myanmar",
        "fun",
        "deutsch",
        "entertainment",
        "cervantes",
        "steepshot"
      ]
    },
    "tags": {},
    "content": {},
    "accounts": {},
    "pow_queue": [],
    "witnesses": {},
    "discussion_idx": {},
    "witness_schedule": {
      "id": 0,
      "current_virtual_time": "275437955266788843112853575",
      "next_shuffle_block_num": 18517170,
      "current_shuffled_witnesses": "0000006b6e7566700000000000000000000000617473656a00000000000000006f72736f6761726400000000000061750064656f726767610000000000000000006f736f7574656e000000000000000000000000006774670000000000000000772e68746f6f6d7300007373656e74690000706f79616c630000000000000000000000656972756300000000000000006172746b636f6c62000000000073656431797567656d6f73000000000000333200006e6f69727566000000000000000000006c61646c657800000000000000006b6f7473656b756c006874686d2e73656165687265766972000000000000006472616b2d646f6f67000000000000616d61627469627375610000000000006b6e70646e616c656f7200000000000000006666696c636d697400000000000000000000000078796e6100000000000000006d697365726168700000000000000000",
      "num_scheduled_witnesses": 21,
      "top19_weight": 1,
      "timeshare_weight": 5,
      "miner_weight": 1,
      "witness_pay_normalization_factor": 25,
      "median_props": {
        "account_creation_fee": "0.200 STEEM",
        "maximum_block_size": 65536,
        "sbd_interest_rate": 0
      },
      "majority_version": "0.19.2",
      "max_voted_witnesses": 20,
      "max_miner_witnesses": 0,
      "max_runner_witnesses": 1,
      "hardfork_required_witnesses": 17
    },
    "feed_price": {
      "base": "3.132 SBD",
      "quote": "1.000 STEEM"
    },
    "error": ""
  }

Here's the subset of the response which is actually (potentially) relevant:

  {
    "props": {
      "total_vesting_fund_steem": "196967804.024 STEEM",
      "total_vesting_shares": "403667516864.126525 VESTS",
      "sbd_interest_rate": 0,
    },
    "feed_price": {
      "base": "3.132 SBD",
      "quote": "1.000 STEEM"
    },
  }

props can be attained by calling get_dynamic_global_properties. total_vesting_* are only referenced in src/app/utils/StateFunctions.js and used to convert SP to VESTS and vice versa. I haven't looked into the full extent of their usage, but this is generally useful to have at hand. sbd_interest_rate is only used on the wallet page.

feed_price is attained by calling get_current_median_history_price, and it's only used on (a) the wallet page (to calculate estimated wallet balance), (b) the market (though commented out), and (c) to calculate state.offchain.signup_bonus which I'm not sure is used anymore.

Second course of action

RT5: remove get_state from /user/transfers, replace with get_account() and get_account_history()
RT4: remove get_state reliance on /~witnesses; all that's needed to populate that page is a get_witnesses_by_vote call.
RT3: (optional) remove get_state from /tags, replace with get_trending_tags() call

RT1, RT2 can remain in place for now; hivemind will provide a compatible API for them until we move to new APIs. That will be a separate project.

steemit / condenser