search

cardano-foundation / cardano-wallet

HTTP server & command-line for managing UTxOs and HD wallets in Cardano.
Apache License 2.0
764 stars 214 forks source link

The reported value of pool saturation is incorrect #1980

Closed Anviking closed 4 years ago

Anviking commented 4 years ago

Context

The wallet calculates and shows the saturation of stake pools. This does not affect the ranking, since the non myopic rewards are calculated in the ledger. | Information | - | | --- | --- | | Version | `2020.7.28 (git revision: 351a1b1134fc67b90b16ed6d129bd1e1d509a1cc)` | | Platform | | | Installation | | # Steps to Reproduce
  1. List stake pools

Expected behavior

  1. Saturation is calculated correctly (same way as ledger; as spec specifies. Think it was total ada in circulation, but should be checked)

Actual behavior

  1. We calculate it based on the relative stake from the stake distribution, which is relative to the total active stake. The active stake is less than e.g. the total ada in circulation.

Resolution

QA

Fixed with cardano-node 1.20.0. The saturation is now based on the live stake, and correctly calculated. API docs are/have been updated as part of #2155.

When I checked the saturation field now corresponds to data from pooltool.io and adapools.org, but with some very slight differences https://docs.google.com/spreadsheets/d/1mO4Ll4IdpniuIefOX9GHbKMpQNUhWpGMvgwrgB39thg/edit#gid=0. I checked with Philipp, but we guess their way of calculating the live stake is just somehow slightly different.

bergr01 commented 4 years ago

Any updates on this one, amigos?

PegasusPool commented 4 years ago

With random ranking and hidden saturation level on daedalus delegators might see well saturated pools ranked in top 10 or so. Is this really a MINOR issue?

KtorZ commented 4 years ago

@PegasusPool ranking is only random in the first epochs where there are no historical data to compute a ranking from. As for the saturation, it currently uses the active stake instead of the total stake. So it is incorrect, but it still gives a right idea of which pools have too much stake compared to others. So it's not blatantly wrong and doesn't prevent the software from operating correctly without putting funds or core features at risk. Hence the minor classification.

Note that minor doesn't mean that we won't look into this. It means that we'll look into it after major bugs have been addressed. Plus, I am afraid that in order to resolve this one, we need some feature upstream with the ledger team (because as a cardano client, we do not know nor have an easy way to know the current Ada circulating supply, this depends on how much is taken out of the reserve which depends on information that only the Ledger knows). So, extended one of the local state query with this piece of information seems like a way forward to me.

@Anviking, may you check on Monday with the Ledger team on this topic :pray: ?

PegasusPool commented 4 years ago

Thank you for the explanation :+1: I really appreciate it

Crypto2099 commented 4 years ago

When historical data? There's no way to spin this as "correct" behavior at this point.

image

KtorZ commented 4 years ago

@Crypto2099 what is your point / question exactly?

Crypto2099 commented 4 years ago

At which point is Daedalus actually going to start ranking pools appropriately? This ticket has had no activity in nearly 3 weeks and an over-saturated IOG pool is currently sitting in the number 1 position. We've been constantly hearing the party line that "We don't have historical data yet in order to properly implement the ranking algorithm". Thus, I feel my question was pretty straightforward and direct: at what point (ballpark) do you think we'll finally have the historical data required to accurately rank pools?

KtorZ commented 4 years ago

Daedalus does rank pools appropriately, according to the ranking formula defined in section 5.6 of the Design Specification for Delegation and Incentives in Cardano.

The ranking formula requires to be able to evaluate pools' performances in order to rank them, which seemingly require pools to have started producing blocks. So, since the first block was produced, ranking started to re-organized itself. Today, pools are ranked as they should in Daedalus.

This ticket is however about something completely different which does not affect ranking.

KtorZ commented 4 years ago

Also, you may disagree with the ranking or, have specific questions about why this or that pool are ranked the way they are. In which case I'll redirect you to cardano-ledger-specs and suggest to open a ticket / start a discussions there about ranking.

We've already identified a few things we are quite sceptical about and working with the ledger team to improve some parts of the ranking formula (which is perhaps too punitive for pools outside of the top k).

Proxiweb commented 4 years ago

Saturated pool with 7% fee ranked No 1 in desirability.... There is something wrong obviously.

KtorZ commented 4 years ago

Saturation and ranking are two different things. Pools are ranked based on how much rewards they would provide to a user considering the case where the pool is just saturated. It is a non-myopic (= not short sighted) metric unlike looking at an immediate return on investment.

If we consider a long term view of the system and if everyone was delegating to the top k pools of the system without saturating them, then the profit of all participant would be maximised and the system would be at equilibrium. Reality is indeed a little trickier because people are changing pools for arbitrary reasons every now and then.

Also, because of this issue #1980, some pools are reported as saturated whereas they are not in practice. I recall the last number of delegated stake was around 30% so it's likely off by a factor 3.

Anviking commented 4 years ago

Saturated pool with 7% fee ranked No 1 in desirability

My take on IOG1:

Although, seems a bit odd why pools like P2P and BCSH2 also with high pledge and low fees, are ranked so low. Only explanation I can think of is poor performance. I did see them at the top when the real ranking first became available. Would be interesting to know the apparent performance values used in the calculation, but not sure we can easily check them.

Also FYI, Daedalus doesn’t show it, but you can get the actual non-myopic member rewards (as queried from the ledger) via the wallet api/cli:

cardano-wallet stake-pool list --stake 100000000 | jq ‘map({ticker: .metadata.ticker, nmm_rewards: .metrics.non_myopic_member_rewards.quantity})’
Ok.
[
  {
    “ticker”: “IOG1",
    “nmm_rewards”: 50697
  },
  {
    “ticker”: “IOG18”,
    “nmm_rewards”: 48183
  },
  {
    “ticker”: “SWIM”,
    “nmm_rewards”: 48011
  },
  {
    “ticker”: “EMUR1”,
    “nmm_rewards”: 47884
  },
  {
    “ticker”: “DIGI”,
    “nmm_rewards”: 47629
  },
  {
    “ticker”: “OCEAN”,
    “nmm_rewards”: 47582
  },
  {
    “ticker”: “1PCT0",
    “nmm_rewards”: 47515
  },
...
hodlonaut commented 4 years ago

You can see rough performance metrics here - e.g. https://adapools.org/pool/00beef284975ef87856c1343f6bf50172253177fdebc756524d43fc1

My understanding is that cardano-wallet is displaying data that's not based on live state of the ledger but is snapshot based, so its severely out of touch with reality (e.g. some pools have had 50+ millions delegated to them since start of the epoch).

If someone could explain to me what does: " "metrics": { "saturation": 2.6545748225171883, " returned for IOG1 via cardano-wallet stake pool list API actually means, that'd be much appreciated...

hodlonaut commented 4 years ago

To your 'just a little bit saturated' comment @Anviking - I tried invoking the stake pool API with stake parameter value of 250 million ADA ( curl http://localhost:8090/v2/stake-pools?stake=250000000000000) and IOG1 still got returned first in the results, does that sound like a desirable outcome for a whale delegator looking to stake their unearthly wealth?

{ "metrics": { "saturation": 2.6545748225171883, "non_myopic_member_rewards": { "quantity": 126742997966, "unit": "lovelace" }, "produced_blocks": { "quantity": 307, "unit": "block" }, "relative_stake": { "quantity": 1.77, "unit": "percent" } }, "cost": { "quantity": 340000000, "unit": "lovelace" }, "margin": { "quantity": 7, "unit": "percent" }, "pledge": { "quantity": 5000000000000, "unit": "lovelace" }, "metadata": { "homepage": "https://iohk.io", "name": "Input Output Global (IOHK) - 1", "ticker": "IOG1", "description": "Our mission is to provide economic identity to the billions of people who lack it. IOHK will not use the IOHK ticker." }, "id": "pool1mxqjlrfskhd5kql9kak06fpdh8xjwc76gec76p3taqy2qmfzs5z" },

KtorZ commented 4 years ago

My understanding is that cardano-wallet is displaying data that's not based on live state of the ledger but is snapshot based

That is correct.

I tried invoking the stake pool API with stake parameter value of 250 million ADA [...] and IOG1 still got returned first in the results,

Yes. That is most likely the "less worse" choice one could make under the assumption than one is willing to delegate ALL of these 250M Ada to a single pool. In practice, one would rather split this over multiple pools.

Anviking commented 4 years ago

cardano-wallet is displaying data that's not based on live state of the ledger but is snapshot based, so its severely out of touch with reality (e.g. some pools have had 50+ millions delegated to them since start of the epoch).

I suspect the stake used for the ranking being delayed (i.e. active stake) is true, but that happens in the ledger and not in the wallet. (To me, sounds like a valid suggestion to them to make it less delayed though)

"saturation": 2.6545748225171883, actually means,

The saturation value the wallet returns is currently nonsensical, hence this ticket. But, this does not affect the ranking.

We cannot fix it without https://github.com/input-output-hk/cardano-node/issues/1659. Though in the meantime we should arguably do something to avoid confusion.

hodlonaut commented 4 years ago

My understanding is that cardano-wallet is displaying data that's not based on live state of the ledger but is snapshot based

That is correct.

I tried invoking the stake pool API with stake parameter value of 250 million ADA [...] and IOG1 still got returned first in the results,

Yes. That is most likely the "less worse" choice one could make under the assumption than one is willing to delegate ALL of these 250M Ada to a single pool. In practice, one would rather split this over multiple pools.

"Less worse"? It's arguably the worst decision you can make under the circumstances - if you ignore all pools that are yet to mint a block (so you don't know if they are even up and running yet), just about any other pool will give you better annual ROS/ROI if you delegate 250m to them, no? :)

hodlonaut commented 4 years ago

cardano-wallet is displaying data that's not based on live state of the ledger but is snapshot based, so its severely out of touch with reality (e.g. some pools have had 50+ millions delegated to them since start of the epoch).

I suspect the stake used for the ranking being delayed (i.e. active stake) is true, but that happens in the ledger and not in the wallet.

"saturation": 2.6545748225171883, actually means,

The saturation value the wallet returns is currently nonsensical, hence this ticket. But, this does not affect the ranking.

We cannot fix it without input-output-hk/cardano-node#1659. Though in the meantime we should arguably do something to avoid confusion.

Thank you for another reference to the open ticket on node side, I completely missed it while scrolling through the thread so far. But I am not completely clear exactly what data that node is returning to cardano-wallet currently is causing this saturation 2.6545 for IOG1 - can you clarify please? (Let's hope there is some progress on it - i.e. providing access to live data - the near future from the node team... since that'd make wallet like Daedalus a lot more useful / responsive)

Anviking commented 4 years ago

But I am not completely clear exactly what data that node is returning to cardano-wallet currently is causing this saturation 2.6545 for IOG1

https://github.com/input-output-hk/cardano-wallet/blob/406c35fdcbb53195517279cbcae3eb1c21aa2609/lib/shelley/src/Cardano/Wallet/Shelley/Pools.hs#L370

The relative stake the wallet can access, is relative to the active stake, which makes this calculation nonsensical. (Should have worked if it was relative wrt. the total stake) (NOTE: "But, this does not affect the ranking." still applies)

i.e. providing access to live data

To be clear, additional local-state queries from https://github.com/input-output-hk/cardano-node/issues/1659 would allow us to display more information in cardano-wallet/Daedalus, but it wouldn't make the ranking respond to over-saturation more quickly. I asked about it internally, but if you care/have questions about it, I suggest opening an issue in cardano-ledger-specs (or re-using https://github.com/input-output-hk/cardano-ledger-specs/issues/1823#issue-689388802) .

cybershaman73 commented 4 years ago

Daedalus does rank pools appropriately, according to the ranking formula defined in section 5.6 of the Design Specification for Delegation and Incentives in Cardano.

The ranking formula requires to be able to evaluate pools' performances in order to rank them, which seemingly require pools to have started producing blocks. So, since the first block was produced, ranking started to re-organized itself. Today, pools are ranked as they should in Daedalus.

This ticket is however about something completely different which does not affect ranking.

I see it now, Section 5.6.1 of the Delegation Design Spec: "To determine a pool’s rank, we order pools by decreasing desirability. The most desirable pool gets rank 1, the second most desirable pool gets rank 2 and so on. We predict that pools with rank ≤ k will eventually be saturated, whereas pools with rank > k will lose all members and only consist of the owner(s)"

The design spec is predicting saturation of pools <= k. This is the problem with the wallet ranking, it's derived from a design spec that is failing to account for saturation so early in the Shelley genesis. If k is not increased soon, this prediction made by the algorithm being used by the wallet will come to pass and many pools needed for a larger value of k will have perished. That is unless of course existing SPOs in the top 150 pools just spin up new pools to offset outflow.

In addition, section 5.6.6 Apparent Performance of New Pools, is deriving its factor from what appears to be a moving average over too short a time frame to allow the smaller pools to be fairly ranked. Apparent performance = 1 was only set for two epochs. This may not be enough time to allow smaller pools to reach a rank of desirability that favorably influences a rising change in k parameter. The health of the network is at stake here. If the project believes that k = 150 will serve its goals then no changes are necessary. If project needs k = 1000, then we're in trouble with the way ranking is currently calculated.

KtorZ commented 4 years ago

If k is not increased soon, this prediction made by the algorithm being used by the wallet will come to pass and many pools needed for a larger value of k will have perished. That is unless of course existing SPOs in the top 150 pools just spin up new pools to offset outflow.

I think there's a misunderstanding about what k is and what saturation means. k is indeed a number that is chosen and the saturation threshold is defined as 1/k, it's a relative proportion of the total stake that is also by extension chosen. So, there will ALWAYS be enough pools to welcome the entire active stake. That is true for k = 10, k = 100 or k = 1000000. The size of the cake is the same regardless of whether you make 10 slices or 100, changing k only changes the section of the slices.

I think many people currently treat the saturation as an absolute number, but the saturation is intrinsically linked to k. The bigger k, the smaller the saturation and therefore, the smaller max stake in each pool.

Note that this does not preclude increasing k indeed. k represent the decentralization goal, how many pools do we aim for. There are 1000+ pools at the moment, but most likely not all 1000 pools are "serious enough" to be considered reliable nodes of the network. So the question is really about how many serious pools that can truly support the network are there? Probably not 1000, but probably more than 150 (current k value if I recall correctly). That's a never ending tweaking though :), perhaps in a year from now, k will be 10 000.

Apparent performance = 1 was only set for two epochs. This may not be enough time to allow smaller pools to reach a rank of desirability that favorably influences a rising change

That is a legitimate remark I think indeed. 2 epochs leaves about 10 days, whether or not this is enough to setup an infrastructure and bootstrap a small community is debatable.

If project needs k = 1000, then we're in trouble with the way ranking is currently calculated.

No we're not. Ranking is dependent on k and ranking is tailored to reach an equilibrium of exactly k pools. Changing k will influence ranking accordingly.

cybershaman73 commented 4 years ago

Thanks for your repsonse @KtorZ I appreciate the time you took to elaborate on our concerns.

I see the problem now more clearly. Presently ranking is per design spec section 5.6.1. However, there's also an assumption made in section 5.9 that delegators will always have "ALL" of the information they need to make the best choice about their delegations. This is where Daedalus is failing delegators by allowing saturated pools to rank, in this case, #1. The Daedalus interface needs to have a way to provide the delegator information about the risks of diminishing rewards that come with delegation to saturated pools. In this way Daedalus is failing to follow design spec per 5.9. This needs to be corrected asap.

aaastakepool commented 4 years ago

So while we established that saturation value returned by cardano-wallet is broken and needs fixing (pending changes in downstream), I don't see any clarification as to why a saturated pool is getting highest non myopic reward value / first ranking. I heard something about 'things are actually implemented according to specs and can be turned on with some flag but are currently awaiting some amount of historical data to be collected' - is there any truth to that, can someone please point me at the code pertaining to such a flag / code waiting for more than X epochs of data to be collected? Why does anything need to be turned on to return a poor non myopic reward score and thus ranking for a pool that is saturated and delegator is hypothetically wanting to delegate amount of ADA (as an extreme case)? I'm somewhat confused at the moment...

KtorZ commented 4 years ago

I don't see any clarification as to why a saturated pool is getting highest non myopic reward value / first ranking.

Make sure to re-read my answers above. Reformulating: saturation is NOT included in ranking. Pools are ranked by descending non-myopic rewards they would provide if they were just saturated.

That last bit is crucial and I think is what is confusing people the most. Saturation and rank are two distinct metrics. The rank tells you which pools are the most attractive in an imaginary future where each pool would be just saturated (and therefore, providing the best rewards). Saturation is an extra information which tells you whether a pool is "full" or not, and helps you choose which pool you want to delegate from the top k.

If every player were rationale and playing in sequence, the system could yield max rewards if people were delegating to unsaturated pools according to their ranks (if the 1st one is saturated, delegate to the 2nd one, and so on). A pool being saturated does not make it less attractive for all players, but it certainly does for new players looking for delegation. Ideally, I think UI should make it possible to gray out saturated pools to discourage folks to delegate to them.

edit: looking back at the ranking formula, an over-saturation would actually have a small effect on ranking as it would slightly decrease non-myopic member rewards but not in a significant manner. That is perhaps something to look at to make more punitive.

Anviking commented 4 years ago

I heard something about 'things are actually implemented according to specs and can be turned on with some flag but are currently awaiting some amount of historical data to be collected'

The full ranking is active.

(During the first — I think 2 or 3 — Shelley epochs the ranking was totally random because pools hadn't started producing blocks yet. I think in an ideal world the ranking would have worked during this time, but…)

On oversaturation: The non-myopic member rewards do take over-saturation into account. The same (capped) rewards will be distributed over more members, so each member will receive smaller rewards. The ranking knows about this.

When looking through the formulas in the spec it's perhaps a bit subtle to see, but I confirmed with Philipp that it is indeed the case.

On: Why is IOG1 so high? My suspicion is still that the "penalty" for over-saturation totally works, but is based on the active stake, and thus just a bit delayed. We did recently see IOG1 drop from rank 1 to rank 7 (Using my stake, but believe most others see the same thing). Its active stake is still only 239.44m, compared to 220m (if that still is the saturation point). I suspect it will drop even more tonight.

I've raised this question to to Philipp, who'll hopefully come back with more clarity on this though!

PhilippeLeLong commented 4 years ago

Saturation is indeed included in the non-myopic member rewards. If all pools were close to saturation and were of equal performance, oversatured pools would be last in the ranking. Assuming enough pools are performing at close to 100% ther should never be an oversaturated pool among the top pools. You can actually see that the ranking is not working properly when using the slider to set the intended amount of stake to delegate. If you pull it up to 200 million, only stake pools with very low saturation and good performance should be among the top ranks. It's also stated in the design specs that the data required for ranking should not be snapshot based, but instead based on live data calculated from confirmed UTXOs. Pooltracking sites like pooltool or cardanoscan apparently have successfully implemented this solution, as they do show the live stake.

From the latest design spec (https://hydra.iohk.io/build/3744897/download/1/delegation_design_spec.pdf, chapter 4.3):

The cost, margin, and pledged stake will be taken from the most recent stake pool registration certificate of the pool. The performance of the pool will be taken as the moving average, as described in Section 5.6.5. The stake of the pool, and the amount of stake that the owners of the pool contribute (in order to check whether their pledge is honoured) is taken from the current stakedistribution (calculated from the current UTxO set and delegation relation on demand when the wallet performs the ordering). The member stake t in Equation (3) is taken to be the stake that the user is about to delegate.

piotr-iohk commented 4 years ago

lgtm.

Also, saturation is roughly inline with the ones from adapools.org. Taking randomly few pools from different parts of wallet ranking(top to bottom).

pool wallet adapools
SECUR 0.63 0.66
ZZZ 0.89 0.87
TRUE 0.037 0.037
CENT 0.0045 0.0045
FORK 0 3.3327770656714e-6
PhilippeLeLong commented 4 years ago

lgtm.

Also, saturation is roughly inline with the ones from adapools.org. Taking randomly few pools from different parts of wallet ranking(top to bottom). pool wallet adapools SECUR 0.63 0.66 ZZZ 0.89 0.87 TRUE 0.037 0.037 CENT 0.0045 0.0045 FORK 0 3.3327770656714e-6

This is not the pool saturation though, but the ratio of the pool's controlled stake in comparison to total stake. Pool saturation is not shown at all in the wallet.

cybershaman73 commented 4 years ago

The point of this issue is to resolve failure of the wallet to follow design spec per section 4.3 and 5.9. In those sections, it is made clear that the wallet has a primary function, in addition to delegation, of providing the information potential delegators need to make informed choices. If a delegator can not CLEARLY see a pool is saturated, either active or live, the wallet is failing to protect them from lower rewards. Ranking pools that are at or near saturation at the very top of the ranking tab drives centralization, the exact opposite of what the wallet is designed for.