paritytech / polkadot

Polkadot Node Implementation
GNU General Public License v3.0
7.12k stars 1.58k forks source link

minimum nomination limit on Polkadot: Status of the off-chain election fix. #2418

Closed pmensik closed 3 years ago

pmensik commented 3 years ago

I am creating this issue because I asked multiple times in various chats, unfortunately without any reply.

I would like to ask what's the status with fixing the off-chain nomination distribution so there's no minimal stake for nominators (it's a bit above 100 DOT at the moment). This is becoming a very serious issue because it prevents a lot of people to stake DOTs because the minimal amount has become about 2,500 USD - I am afraid this makes Polkadot almost a "permissioned" network. I created this issue as a "last resort" since I couldn't get the answer anywhere else - if this is being worked on and when we can expect it to be done. Thanks a lot for the insight!

kianenigma commented 3 years ago

My short answer is keep an eye on these PRs and how they are deployed:

https://github.com/paritytech/substrate/pull/7908 https://github.com/paritytech/substrate/pull/7909 https://github.com/paritytech/substrate/pull/7910

These works will optimize the whole process, allowing potentially many times more nominators to be active. Moreover, I was planning on tweaking some parameters to see if we can allow more nominators (similar to this one) for the next release as a temporary mitigation in favour of Polkadot's stakers.


If you are interested in the matter a bit more:

You might ask why we have this issue in the first place? The answer is that nomination is different from (so called) delegation in the sense that you are not merely replicating/supporting/backing the behaviour of someone else, but rather imposing your own direct opinion on the system as well. This is why typically when you delegate, you name 1 entity (e.g. a validator) whom you wish to support. Whilst in nominating, you provide a list of preferences, and an election process needs to process all of these lists to come up with an optimal set of validators that optimizes a set of goals (which course, serve toward decentralization). At this point it should be clear the supporting an arbitrary large number of nominators is simpy not trivial in a system with constrained resource like a blockchain. This was explained in the original blog post here.

The good news though, is that with Polkadot's sharded design, we are very likely to be able to radically change this, and improve the standard of nomination in the ecosystem, in the foreseeable future. The truth is, as long as the relay chain is busy doing everything for everyone, it is unlikely to be able to allocate a lot of time/blocks to the election processing. The long term vision, however, is to move the entire election process to a separate system parachain. Then, we can safely process a massive number of validators and nominators directly on-chain. Until that happens, we are limited to optimizations such as the above PRs.

As a side note, we've also considered adding the ability to delegate on-chain, as well, next to nomination, to allow smaller staked people to replicate the behaviour of other nominators and still share their rewards. But this will add a lot of complexity, and feels more like taking a step back rather than forward. Moreover, given the plan for a system parachain in the near feature, and other easier optimizations being possible, delegation is likely to be an approach that will neither benefit nominators, nor be reasonable to implement in a timely manner.

But a more worthwhile effort is to move everything toward multi-block processing, and this is indeed the direction of our efforts at the moment. A system parachain that only does election-related work has all the time in the world (quite literally) to process a large number of nominators, it just needs to be smart enough to split the work into multiple blocks. It is very likely that we will start experimenting with multi-block elections even on the relay chain in the future, but that will partly depend on when parachains become available.

This was a brief overview of the technical side of things. Lastly, I want to clarify:

This is becoming a very serious issue because it prevents a lot of people to stake DOTs because the minimal amount has become about 2,500 USD - I am afraid this makes Polkadot almost a "permissioned" network.

I don't expect everyone to know the system in such detail, but since you've asked:

If you nominate with a small amount of DOT, your opinion is actually reflected on choosing the final validator set. The election process is done offchain, and sent back to the relay chain by validators with a set of validators that reflects all of the nominations. Once being sent back to the chain, for the purpose of rewards and slashing bookkeeping, we only trim the smallest staked nominators, to make the solution feasibly processable in one block. Nonetheless, the winner validators do not change based on this trimming, which effectively means from a "centralization of power" point of view, Polkadot is utmost decentralized (even in its rather unoptimized, novice form right now).

Furthermore, recall that all democracy operations are open to be participated therein, without any minimum DOTs (other than existential deposit needed to have an account in the first place).

With that, I want to emphasize that this "minimum DOT to stake" is about reward and slash only, not the actual nomination power that influences the validator election, which is what nomination is all about, from a governance point of view: choosing trustworthy validators based on the average network opinion.

Nonetheless, it is reasonable to assume that with all the plans noted above, Polkadot will allow more and more (reward-able) nominators as it grows. Again, keep an eye on the above PRs and I hope improvements come soon.

paxelpixel commented 3 years ago

this is a super interesting read! thank you for explaining the issue and providing insights into the solutions Kian!

pmensik commented 3 years ago

@kianenigma Perfect answer, thanks a lot! That's much more than I hoped :-)

Benny739 commented 3 years ago

I saw the minimum stake dropped from 187 to 65 and now 68. Are there any news? What will happen if I have for example 100 DOT and the system splits it 50/50 to 2 validators. Will I then get rewards if the limit is 68 or not?

kianenigma commented 3 years ago

I saw the minimum stake dropped from 187 to 65 and now 68. Are there any news?

A runtime upgrade went in and while I don't have anything off the top of my head, there might have been some optimizations applied that would improve this. The next upgrade will probably have a similar effect.

What will happen if I have for example 100 DOT and the system splits it 50/50 to 2 validators. Will I then get rewards if the limit is 68 or not?

You will get reward as long as you have active nominators (from the perspective of this bottleneck). The only other perspective will be oversubscribed validators, meaning that if one has more than 128 nominators, only the top 128 can claim rewards.

But the scenario that you explained can never happen. The metric of minimum staked nominator from polkaview comes from simply iterating over all active backings of nominators to validators, and taking the min of that. So you can't have 50 when the minimum is 64.

Benny739 commented 3 years ago

Thanks for your explanation. Are there also any plans to increase the 128 nominators? I saw it was already at 256 and got reduced to 128.

kianenigma commented 3 years ago

I am not aware of any ongoing plans, but the governance can propose a change to this at any time rather easily, so maybe you can discuss it in the Polkadot/Kusama direction chat.

To the best of my knowledge we can easily bump tot 256 and it was prevented just to take it one step at a time. cc @shawntabrizi

andr3 commented 3 years ago

@kianenigma Thank you for the detailed answer. I took several passes and re-read your comment multiple times to make sure I fully understood the work being done and the implications of such a solution, to the best of my ability.

I'm very new to the technicalities of Polkadot, I must admit. I have only started staking a mere 13 days ago. I was negatively surprised when I learned I was not eligible for rewards AFTER starting my staking period (12 days in). I was expecting rewards every two days... kept waiting and checking, until I found the October article you linked.

I'd like to share a few thoughts only in the interest of openness and feedback from a real world user. Despite me being technically minded, (trained in computer science) I have dabbled in User Experience and hope these words help finding the better way forward and not just frustrate the amazing team behind such a great project.

Some notes:

The comments:

My feedback, please expand > If you nominate with a small amount of DOT, your opinion is actually reflected on choosing the final validator set. From the common user perspective, locking down my DOTs without any rewards, even considering my nominations matter to the election, in the current crypto landscape feels like a bad investment decision. I'm drawn to move my DOTs to an exchange and stake them there, which apparently don't have a limit? Whether they'll generate rewards if below the limit or not, that's left to be seen... I haven't tried that, yet. Am reluctant to do so, too. This has the negative side effect that I will not be in charge of nominating anyone (as far as I understand, binance won't give me that level of control) nor will I be able to use my DOTs for any governance voting. > The answer is that nomination is different from (so called) delegation in the sense that you are not merely replicating/supporting/backing the behaviour of someone else, but rather imposing your own direct opinion on the system as well. On this... my first experience on selecting nominators was very... opaque. I was given very little information from the validators and given very few hints of how to make that decision fairly. You mention my opinion is put into the system... but that is an uneducated opinion for first-comers/not-technically savvy (and I don't mean financially, I mean engineering-wise). If you require a DOT holder to do heavy due diligence on validators with very slim information, users will either walk away or pick some that simply "look good". Giving a weak value of confidence in the opinions being put into the system. Two more bits of feedback, hopefully useful: - Upon nominating/staking, I was not given any indication my stake was below the minimum nomination. Frustrating. - If the minimum nomination level is variable through time, the system is not reliable to stake long term. - In some eras I will get rewards, in others don't -- even when no slashing occurs. (affecting the expected annual percentage yield forecasted. - This will draw me to other chains with more reliable rewards systems, I'm afraid. Speaking from a financial perspective. - Unbounding the amount staked has forced me to wait 27 days to have access to the funds, leading me with a sour taste in my mouth for locking my money without any rewards. If in this period, I won't get rewards even if I am over the minimum level, this is extremely disturbing. - Quite possibly I made a wrong decision somewhere in the process to now have to wait these days... Mea culpa. It was very hard to navigate the process of staking, tbh. The rewards part is an essential part of the whole system. People are selfish. If they're locking down funds and getting no rewards, even if the validators didn't misbehave and no slashing occurred, they feel duped. Won't stake again on DOT and moved away from the ecosystem. Which nobody here wants to see.

☝️ Sorry for the lengthy comment. I apologise for not presenting any solution, as I don't feel equipped with enough knowledge to do so, at this point.

However, I will say that the reality of multi-block processing does seem to be very promising.

Thanks a lot and keep up the great work.

yuzmemak commented 3 years ago

Hi @andr3, you are touching many good points, dot developers should listen you.

Especially 27 days issue makes dot ugly. Public will not forgive such a old tricks, nearby minimum dot level for staking.

I see what dot developers aim with these 27 days. May be expectable for stable price and feature but during this 27 days, staking should be continue, at least. In this perspective, ada project has more freedom. We will see that many dots holder migrate to ada during next eras.

Benny739 commented 3 years ago

Any ETA on the fix? The min DOT amount is growing rapidly and with the 27 days lockup a lot of smaller holders will be screwed soon because their dots are locked and they are also not getting any staking rewards.

kianenigma commented 3 years ago

@andr3 apologies for the delay. Here are my thoughts on your comment.

From the common user perspective, locking down my DOTs without any rewards, even considering my nominations matter to the election, in the current crypto landscape feels like a bad investment decision. I'm drawn to move my DOTs to an exchange and stake them there, which apparently don't have a limit? Whether they'll generate rewards if below the limit or not, that's left to be seen... I haven't tried that, yet. Am reluctant to do so, too.

This has the negative side effect that I will not be in charge of nominating anyone (as far as I understand, binance won't give me that level of control) nor will I be able to use my DOTs for any governance voting.

I agree that this is not an optimal investment, at the moment and for those who are below the threshold. I can't comment further on the examples you named as I am not familiar with the details, but your opinion sounds fair to me. This is why we are trying to maintain a balance of: while the users are free to move to different offchain staking solutions, we do our best to keep a portion of them incentivized to stake on-chain for better decentralization and stake distribution. Ergo, the planned fixes and the continuous efforts to improve this.

was given very little information from the validators and given very few hints of how to make that decision fairly.

Through which ways did your try to inform yourself? This is potential issue, but indeed not a technical one and more about communication and education. I hope community projects will pick up that will analyze validator behavior and provide transparent reports. I recall seeing such projects being discussed in the governance chat of polkadot, but haven't followed them myself. my TLDR is that this is a good point, but not one that belongs in a technical issue, and would be treated well in a governance or community chat, or as a RFP.

You mention my opinion is put into the system... but that is an uneducated opinion for first-comers/not-technically savvy (and I don't mean financially, I mean engineering-wise).

But note that nomination is, by design, more complicated than, say delegation and the average user is expected to spend some time studying the system, inspecting candidates and making a good decision. After all, there is a risk in nomination as well, so the nominator is expected to not act in a very naive manner, and invest some time. All that being said, I wanted to argue that some technical jargon in the discussion, while hard for the first-time user, is beneficial+necessary in the long term.

Upon nominating/staking, I was not given any indication my stake was below the minimum nomination. Frustrating.

Value is dynamic, so what is shown now is not the same as tomorrow, but yeah.. I get your point. I won't make an issue for it in https://github.com/polkadot-js/apps/issues/new yet, because I am sure in the next 1-2 upgrades we will increase this capacity a lot and the limit will become basically pointless, but feel free to discuss it there if you wish so.

If the minimum nomination level is variable through time, the system is not reliable to stake long term.

I can't provide reason why you should (or should not) stake in which chain, so I will refrain from answering this directly. I've tried to be as transparent as possible about the current state of affairs in this discussion, explain why this limit exists and how we think it will be solved (with linked PRs), and the rest is outside the scope of what I can and should comment on.

Unbounding the amount staked has forced me to wait 27 days to have access to the funds, leading me with a sour taste in my mouth for locking my money without any rewards. If in this period, I won't get rewards even if I am over the minimum level, this is extremely disturbing.

The 27 waiting day is unavoidable and a very important security measure to protect against certain attacks and vulnerabilities of PoS. I totally understand that it is annoying, but arguably it is for the benefit+security of the overall network, which includes you as well at the end of the day as DOT-holder.

kianenigma commented 3 years ago

Any ETA on the fix? The min DOT amount is growing rapidly and with the 27 days lockup a lot of smaller holders will be screwed soon because their dots are locked and they are also not getting any staking rewards.

Runtime version 29 includes the new pallet that will fix this, but it is not active yet. That is planned for the next release, 30. Should things go well, I expect this to be solved by the upgrade to version 30. You can follow this here: https://github.com/paritytech/substrate/pull/8113, and all the related PRs are also linked there.

andr3 commented 3 years ago

@kianenigma Thank you so much for your thoughtful response, I totally understand the delay don't worry about it. I appreciate some of the topics raised here are not relevant for a technical issue and apologies for going on a little rant, back then. ;)

I'll investigate the issue tracker/RFP you pointed to and try to formalise a proposal to address some of the non-technical aspects. Thanks for directing me, truly appreciate it.

The 27 waiting day is unavoidable and a very important security measure

Oh, 100%. I totally understand and agree with having some locking period. It was only especially frustrating given I didn't earn any rewards contrary to expectation, so therein lies the source of the problem—the lack of rewards.

Through which ways did your try to inform yourself?

Fair question. :) I believe I started on polkawallet and then used polkadot portal to inspect further. I was indeed aware of the possibility of slashing. Will keep an eye out of community projects to make this investigation clearer and easier.

Anyway, great to hear we have movement in this front. Wishing you and the project all the best! 💪

yuzmemak commented 3 years ago

@kianenigma

The 27 waiting day is unavoidable and a very important security measure to protect against certain attacks and vulnerabilities of PoS.

When we respect this 27 days security measure, why not to keep them as nominator until end of 27 days? As far as I understand these dots have been using as stake power.

kianenigma commented 3 years ago

When we respect this 27 days security measure, why not to keep them as nominator until end of 27 days? As far as I understand these dots have been using as stake power.

One of the reasons for the 27 day limit is that you can be responsible for your actions as validator/nominator for at most this amount of time in the past. Therefore, what you are proposing is not possible. From the moment that you intend to not nominate, you must actually not nominate but stay bonded in the system for 27 days, and then you are free to go. The point of this is that in the 27 days, a slashing event of the past might be unravelled by a reporter and your stake needs to be available for slashing. If you keep nominating, this cycle will just repeat indefinitely.

yuzmemak commented 3 years ago

When we respect this 27 days security measure, why not to keep them as nominator until end of 27 days? As far as I understand these dots have been using as stake power.

One of the reasons for the 27 day limit is that you can be responsible for your actions as validator/nominator for at most this amount of time in the past. Therefore, what you are proposing is not possible. From the moment that you intend to not nominate, you must actually not nominate but stay bonded in the system for 27 days, and then you are free to go. The point of this is that in the 27 days, a slashing event of the past might be unravelled by a reporter and your stake needs to be available for slashing. If you keep nominating, this cycle will just repeat indefinitely.

I see but now what I worry that how alternative POS project handles this issue? They have no such a time for potential slashing.

But same time they have less APY than dot. This 27 days explain that how dot keeps 13% APY when others around 5% APY with same inflation. These 27 days increase active nominators income.

Now time to read others project slashing protection policies. Because, minimum dots for staking again reach 200 dots level. There will be no feature here for poor's guys in polkadot.

kianenigma commented 3 years ago

For anyone interested, if you like to hear more details about his, you can vote my candidate talk in the upcoming polkadot decoded :P

Screenshot 2021-03-20 at 09 38 25
horaciob commented 3 years ago

at the end... if it forces to us to leave our Dots on a centralized exchange what do you get? (centralization)

PD: a validation where you cannot lock your Dots if you are not going going to get rewards would improve a bit our user experience... 6 eras waiting rewards + 28 days to unbound... is too much. there is no excuses to not improve user experience.

xlc commented 3 years ago

Just to keep in mind that there are multiple parachains are working on staking derivatives to enhance staking experience. So the Polkadot layer can just offer a secure & solid base layer and parachains can build something on top of it to offer better user experience.

horaciob commented 3 years ago

there is no excuses to have a good experiences in every layers. Why not validate? this is about the trust.

And everybody tell me the same hold in kraken and you will get your 12%, at the end all "minor" holders will concentrate all their dots on a big exchanges who will get a Hugh amount of dot for getting more voice on the governance (where is the decentralization?).

I don't know maybe i misunderstood something.

kianenigma commented 3 years ago

there is no excuses to have a good experiences in every layers. Why not validate? this is about the trust.

There -- most certainly-- are technical reasons for it that you can find in this very page (toward the beginning). I'm happy to discuss them further if you have any follow up comments to any of the explanation that I provided earlier, and have already mentioned that there are works both on the relay chain, and as @xlc mentioned, on certain parachains to provide a wider and more appealing nomination options.

TLDR; because of the way that NPoS needs to execute and validate an expensive election algorithm (as opposed to ranking by stake etc.) it is simply not trivial to support an infinitely large number of nominators. Things need to fit into blocks, blocks need to be within 2 seconds, and all other sorts of limitations exists.

horaciob commented 3 years ago

Thanks @kianenigma this post that you mention explain very well your scaling problem.

But, I guess that maybe you can have 2 workaround for this,

1- If you calculate the minimun bonding in every era you would validate new stakers against that number (if you know that in the last era you need 203 dots to stake you could know that me with 100 dots are veeery far away that number, maybe you can accept a new staker with 200 dots but warning them)

2 There is any reason that people who didnt receive reward would wait 28 days to unbond?, would be fair that at least they have not to wait 28 days after no benefits for staking

kianenigma commented 3 years ago

2 There is any reason that people who didnt receive reward would wait 28 days to unbond?, would be fair that at least they have not to wait 28 days after no benefits for staking

On paper: as I noted above even when you are not receiving rewards, you do impose your opinion on a validator and do help them get elected, so you are still responsible for slash if they misbehave, so you still need to be bonded for 28 days. That being said, we could provide a way around this. But my main efforts is mostly toward allowing more nominators at the moment. I'll make an issue for it in substrate.

The first comment, if I understand it correctly, is also discussed here before and is a UI matter (not the polkadot protocol itself) and perhaps wallets and apps can warn users about this, but for that you need to pursue it in those repos, not here.

horaciob commented 3 years ago

Thanks @kianenigma for your reply and to be open to listen me, now, i know a bit more about polkadot and that's good, and give me a confidence on the people who is behind the project I will still supporting this project.

pmensik commented 3 years ago

Hello @kianenigma, I am looking at the release notes and it seems like the fix for this issue is not there (also, the minimal stake keeps rising even though the new release is out for a couple of days). Can you please tell us what's the status? Thanks a lot

kianenigma commented 3 years ago

yeah, so it is now just a matter of waiting for the voting process to be completed. Usually the technical committee fast tracks runtime upgrade proposals, so I think we should be at most a week or two away. I believe in Kusama runtime 30's proposal is already approved by the council.

davidurco commented 3 years ago

@kianenigma Is it known what will be the new min. stake?

kianenigma commented 3 years ago

@kianenigma Is it known what will be the new min. stake?

I think 0. With the new system we can support up to more like 50-100k active nominators and currently have less, so no limit hopefully exists for some time, depending on the network growth.

Benny739 commented 3 years ago

yeah, so it is now just a matter of waiting for the voting process to be completed. Usually the technical committee fast tracks runtime upgrade proposals, so I think we should be at most a week or two away. I believe in Kusama runtime 30's proposal is already approved by the council.

Any news about the ETA?

kianenigma commented 3 years ago
Screenshot 2021-04-27 at 08 33 00

The upgrade is now in the external queue (already approved by the council). I think it should land by the end of the week. You can follow up or ask further in the polkadot watercooler.

Benny739 commented 3 years ago
Screenshot 2021-04-27 at 08 33 00

The upgrade is now in the external queue (already approved by the council). I think it should land by the end of the week. You can follow up or ask further in the polkadot watercooler.

Thx :) I know this is OT, but I couldn't find an answer. How many DOT do I have to keep transferrable (when staking) that my account doesn't get deleted? Can I stake everything and just keep 0.1 DOT transferrable or do I need at least 1?

aelesbao commented 3 years ago

@Benny739, for Polkadot you should keep at least 1 DOT in your wallet, and for Kusama, it should be at least 0.001666 KSM. This is called an existential deposit and you can find more info about it here.

Benny739 commented 3 years ago

@aelesbao thx, do the staked DOT count towards the existential deposit? So is it about the complete account balance or the transferable balance that needs to be at least 1?

horaciob commented 3 years ago

I made a small test, before send all my dots, with 6 dots on 4 validators and after 2 days, all validators were active but there is no reward for me.

Do you know when this will be updated?

hirschenberger commented 3 years ago

I made a small test, before send all my dots, with 6 dots on 4 validators and after 2 days, all validators were active but there is no reward for me.

Do you know when this will be updated?

On Block #4874334 (~1day) https://polkadot.polkassembly.io/referendum/22

kianenigma commented 3 years ago
Screenshot 2021-05-02 at 17 37 41

Note that we might hit such bottlenecks again sometime in the future, but at least for now the situation is resolved.

Note that there are handful of validators who are oversubscribed, but that's a totally different story.