Staking: Controller Deprecation Step by Step & Tracking Issue

This is an issue for documenting and tracking the status of controller deprecation. With https://github.com/paritytech/substrate/pull/14039/ and https://github.com/paritytech/polkadot-sdk/pull/2380 now merged (the latter awaiting deployment) we can now consider the strategy to completely deprecate controllers. Below is a proposed step by step plan of doing so.

Current state of unique stash controller pairs

The following stats reflect how many bonded pairs there are on Polkadot and Kusama, how many of them are unique (a different controller and stash account), as well as how many of those are actively staking when the stats were generated.

For those who wish to reproduce these statistics for the current era, I have uploaded rossbulat/polkadot-helper-scripts. Delete the included json files and run yarn controllers to re-generate the statistics.

// Generated on 25/05/2024 09:05 UTC

Polkadot:
Total records:  63953
Unique pairs:  4848
Percentage still unique:  7.58 %
Unique pairs that are active:  4082
Percentage of unique pairs that are active:  84.20 %

Kusama:
Total records:  26514
Unique pairs:  4992
Percentage still unique:  18.83 %
Unique pairs that are active:  4642
Percentage of unique pairs that are active:  92.99 %

Westend:  0 unique pairs / migration completed.

Deprecation Step by Step

The following steps propose a route for bringing unique pairs to zero, whereby the staking team could then focus on pallet-side removal of the controller without disrupting user experience.

Pallet side

1) Add a temporary deprecate_controller_batch Staking Admin origin call to the staking pallet, that takes a Vec<AccountId>, to safely move controllers over to the stash. This call should check whether the pair is still unique at the time of execution, and ignore if they are not. @gpestana @Ank4n does this sound reasonable?

2) Set up OpenGov referendums to execute deprecate_controller_batch calls. (can use polkadot-helper-scripts to scrape the current unique pairs). The amount of accounts provided, & calls needed, will depend on call weight.

Notes:

Step 2 should be executed once step 1 has been deployed in a runtime upgrade.
UIs can be notified of the upcoming changes once 1. is merged & the runtime upgrade block is confirmed (DOT and KSM).

Application side

3) Provide a prompt on the staking dashboard overview page stating that controller accounts are now deprecated, and follow if you wish to continue using your controller account as a proxy.

4) Update Polkadot Wiki outlining the deprecation and link to proxy accounts documentation to continue using your controller as a proxy.

To ensure minimal disruption, 3, and 4 should be deployed together as 2 is enacted. This strategy assumes that stakers have not discarded their stash keys, which are needed to register a proxy account with their deprecated controllers.

Alternatives to OpenGov

A permissionless lazy migration approach could also be taken to move controllers to stashes. The benefit of OpenGov however is community inclusion where the community can have their say in the deprecation by voting, as opposed to the negative perception of the Polkadot devs making all the decisions. From a social perspective, the referendum also gives more publicity to the deprecation and increases the probability of users moving their controllers over to their stashes manually before the referendum is enacted - the lazy migration approach wouldn't benefit from any publicity on Polkassembly, Subsquare etc.

Potential Problems

Users report they no longer have access to their staking accounts

In the event users are confused about the update, most likely resulting in their stash account not being imported in a wallet, Polkadot.network support and the staking dashboard Canny forum are both available to offer guidance and support. This scenario will most likely occur initially if users attempt to claim payouts, and those users are likely to be validators who will likely opt to set up a proxy (if they haven't done so already).

Users actually no longer have access to their staking accounts

Loss of stash keys should not be the responsibility or concern of Polkadot. The staking pallet currently does not contain root origin calls whereby a referendum could force a nominator to unbond and withdraw. To my knowledge, 2 referendums of a) unbonding, and b) withdrawing + setting a proxy for the lost stash) would be the only way to circumvent such a scenario (maybe @kianenigma can correct).

Add a temporary deprecate_controller_batch root origin call to the staking pallet, that takes a Vec, to safely move controllers over to the stash. This call should check whether the pair is still unique at the time of execution, and ignore if they are not. @gpestana @Ank4n does this sound reasonable?

Set up fellowship whitelisted OpenGov referendums to execute deprecate_controller_batch calls. (can use polkadot-helper-scripts to scrape the current unique pairs). The amount of accounts provided, & calls needed, will depend on call weight. These fellowship whitelisted calls have a less stringent support curve (@joepetrowski have whitelisted calls generally been enacted quicker in recent history?).

Whitelisted calls can go faster than Root, and in general have. Root can also only hold one referendum at a time, while Whitelisted can have 10 IIRC. There is also a StakingAdmin track though, why not use that?

Whitelisted calls can go faster than Root, and in general have. Root can also only hold one referendum at a time, while Whitelisted can have 10 IIRC. There is also a StakingAdmin track though, why not use that?

Ah good idea, less decision deposit and max deciding should be plenty enough to carry out the needed calls.

Update Polkadot Wiki outlining the deprecation and link to proxy accounts documentation to continue using your controller as a proxy.

Is it possible to migrate controller accounts as proxy to stash? That would mean no functional change for users, right? Thinking out loud, a permissionless call could remove controller from staking pallet and set up the controller account as proxy for the stash account with staking permissions. For these migrated set of users, we should be able to create proxies for free this way as well. Wdyt?

Is it possible to migrate controller accounts as proxy to stash?

Do you have any thoughts on how to handle the proxy deposit? We could cover it but it is not as a trivial amount as paying for call fees, etc.

permissionless call could remove controller from staking pallet and set up the controller account as proxy for the stash account with staking permissions.

I agree with @rossbulat in that I'd prefer any batch changes to the state of the stash/controller to go through governance, instead of adding a permissionless/ migrations for this type of migration. These migrations change account permissions so better to be as transparent with the community as possible.

Update Polkadot Wiki outlining the deprecation and link to proxy accounts documentation to continue using your controller as a proxy.

Is it possible to migrate controller accounts as proxy to stash? That would mean no functional change for users, right? Thinking out loud, a permissionless call could remove controller from staking pallet and set up the controller account as proxy for the stash account with staking permissions. For these migrated set of users, we should be able to create proxies for free this way as well. Wdyt?

If the proxy account economics were not broken by doing this, then it would make sense. But as it stands there would be an additional 5,000+ proxies registered on both Polkadot and Kusama, with no incentive to remove them. We already saw that ~10% of stakers opted to use a separate account for their controller. I also suspect many who did set different controller accounts did so because the Wiki stated to do so, quite strictly, and JS Apps forced the convention in the UI. So overall the demand for proxies is conceivably pretty low in reality.

An opt-in approach post migration is the optimal strategy imo, with full transparency in the Wiki and a dismissible staking dashboard prompt that will show for every visitor for a period of time.

@rossbulat Could you please address my concerns as a nominator regarding transition during this change:

1.) For a nominator, will it be needed to unbind the stake (and wait 28 days)? If yes - can it be implemented without this requirement? As a nominator - I can't image such requirement will be accepted positively by the participants (lost income), not to mention massive drop in economic security of the network until the transition ends.

2.) Will the transition be done in lockstep with the developers of Polkadot apps running on hardware wallets? In particular, Polkadot Ledger app never worked with stash account, and only worked with controller account set as a target for receiving of the rewards... I assume similar limitations could be present in Polkadot apps on other hw wallets and staking apps. Will there be enough time for these developers to catch up, update the software and communicate the needs so that the transition is without problems?

3.) Will there always be a way to access the stake for users who don't actively watch Polkadot development? Imagine a user who learns about these changes a year after the transition finishes. If his/her stake is removed from a nomination, that's a bummer, but IF their funds became lost/inaccessible, without any means to migrate them way later, that would be a catastrophic for the public's image of Polkadot, and also for the users who will lose a lot of money. That can't happen.

4.) Iterating on point 3.) - you can't move the responsibility for (having a) stash account to the users, and declare their DOT lost forever if they don't have stash account recovery written down. As written in point 2.) - Ledger (the main HW wallet used for staking Polkadot - for better or worse) never worked with it, and therefore users simply don't care about the stash account - either they don't have it at all, or they lost recovery info.

5.) Bringing publicity to this change via governance - average users don't even know that there's some governance, and if they know, they don't participate in it. And it shouldn't be required to participate in it for staking.

The ideal way for users (nominators) and, at the end - developers - is that this transition happens automatically:

without users intervention, turning the controller account to stash (with controller like ability)
without stake unbind
without need to have a separate stash (if you turn the controller to the stash account, you get the stash account you need, to which the users have guaranteed access)
without panics in community (and pretty bad publicity) about losing access to lots of money
without Parity being bombarded on support channels by angry users to fix their staking
after this automatic change, users can set up their other stashes/proxies with all the flexibility they want

Since everybody can participate in Polkadot anonymously, you don't have any communication channels with nominators. (Even if you did, they may not listen). There will always be people left behind, and for those, you will still need to have the migration path from (even 10 years old) forgotten controller to a new stash implemented in code - you don't want that liability (for code maintainability).

Ultimately, this all seems like a technical change, a minor step in evolution of Polkadot. I don't think that users care about this change at all.

To let the users be exposed to this technicality - there are only downsides (potentially catastrophic ones for users, and for Polkadot publicity).

It makes perfect sense to use governance voting for reaching consensus on major things (like inflation, apr, ...) but using governance for this is pretty much off. Too small granularity of the decision making. And negligible publicity from the voting.

There is a way to do this without users intervention, reducing the risks of lost funds to zero (and lost income), tested heavily to cover all edge cases, with guaranteed success for all users, without panicking of community and support backlash. Why forcing the users to take actions to the only plausible direction when you can do that automatically and for users such actions are because of a technical issue they don't understand/care about.

Personally, I'm scared. I already got bit once when staking DOTs, by an algorithm issue which removed me from receiving rewards for months. Being exposed to this change, and being dependent on external actors like Ledger app to act accordingly and in timely manner, gives me goosebumps. And even a slight possibility to lose everything I've invested to Polkadot because of some software issue in complicated transition because of unnecessary users involvement...

1.) For a nominator, will it be needed to unbind the stake (and wait 28 days)?

No - there is no action needed for existing nominators.

Polkadot Ledger app never worked with stash account

Ledger does work with both stash and controller. I did a lot of work with Zondax and Ledger last year to ensure that the necessary staking & proxy extrinsics work for staking. Staking dashboard is fully Ledger supported as a result, and other apps can also submit the required extrinsics using Ledger.

3 / 4) If his/her stake is removed from a nomination...

The deprecation will not touch any stash related chain state. It will only update one storage item, Bonded, which will not effect any staking setup. Post deprecation we will be able to remove the Bonded storage altogether and simplify the pallet logic, which will also make migrating to the staking parachain easier.

Users Should have their stash keys written down. The chain should not be held accountable if users lose their keys. Polkadot is upgradable and changes should be expected. In the controller account's case, less than 10% of nominators were actually using a unique controller to their stash, and this is partly why we chose to go down the deprecation path. That figure is much less now on Polkadot and Kusama, since we turned off the ability of setting unique controllers a good few months ago now.

Can users theoretically recover? Yes - archive nodes would have a record of the controller accounts before deprecation and if needed governance would be able to unbond, withdraw and then transfer funds on a stashes behalf.

but using governance for this is pretty much off

Disagree and this is incorrect because it would be a governance decision in any case; either with the path of StakingAdmin we've opted for, or via a runtime upgrade.

This initiative has been ongoing for over a year now, and it has been around 4 years since the idea was first bought up (when proxies were introduced to the Relay chain). Rest assured this has been well thought out.

Ultimately controller accounts are not what users wanted to use and it is a part of our job to correct missteps in assumptions made along the way. This is a good example of removing an unused feature, decreasing pallet complexity, and allowing proxies to inherit a solid use case.

In regards to apps, we learned a while back that simply leaving controllers in chain storage would not force UX improvements across the board - this is one of those scenarios that the protocol needs to force those improvements to happen; this will lead to consistency between all UIs.

Personally, I'm scared. I already got bit once when staking DOTs, by an algorithm issue which removed me from receiving rewards for months. Being exposed to this change, and being dependent on external actors like Ledger app to act accordingly and in timely manner, gives me goosebumps.

Sorry to hear about the reward issue - it is very frustrating. Rest assured this has been a planned and surgical process. Westend has already carried out the migration, and although it doesn't reflect a chain with real value, there were no technical errors with the migration and staking continued without fault. I'm of the view that Polkadot staking should be improving and evolving to meet the needs of network participants - it already has been subject to several iterations that have dramatically improved the system, such as scaling the nominator limit, and introducing nomination pools. We will soon no longer have over subscribed validators, and of course, things will be simpler by not having to learn the stash and controller abstraction. Upgradability is a big thing we can leverage with Polkadot to keep our staking system relevant, and to ultimately be competitive among other protocols. I think we're making a lot of right calls to head in that positive direction.

Quick update: We are awaiting restore_ledger calls to be executed (to fix erroneous ledger entries) before opening referendums for deprecate_controller_batch calls on Kusama (and then Polkadot). Updates will be provided here as and when these items are actioned.

Sounds good, thanks for the upadte @rossbulat

paritytech / polkadot-sdk