[QA] How will we handle inability / reluctance of Pool Operators to deploy upgraded versions

[mark-stopka]

Can you please add to the Open questions wiki page following question?

Current ITNv1 shows rather significant fraction of pool operators being reluctant to deploy new versions of the node software; how to handle such situation

As of 2020-Feb-25 13:53 UTC, Jormungandr version distribution for those who report their data to PoolTool is as follows:

Version	Nodes	Nodes %	Days since release
0.8.6	3	2.80%	41
0.8.7	1	0.93%	33
0.8.8	1	0.93%	26
0.8.9	63	58.88%	26
0.8.10	0	0.00%	12
0.8.11	27	25.23%	6
0.8.12	12	11.21%	0
Total	107	100.00%

While we currently don't know the stake distribution delegated among these pools, we do know that breaking change has been introduced in 0.8.10 which was later rolled back in 0.8.11, while the only reason to rollback was not that it would effectively create a network fork, it certainly is a thing to take into consideration when ensuring network upgrades finality.

Please also note that 0.8.9 actually consists of 7 different releases with following distribution:

Build	Nodes	Nodes %
46df05c7	26	41.94%
30d20d2	10	16.13%
3b1cdb1b	1	1.61%
87c5bc08	1	1.61%
36772bb1	16	25.81%
30d20d2e	8	12.90%
Total	62	100.00%

30d20d2 has been provided by Jormungandr team, remaining builds are community contributed version with either cherry-picked commits from future releases, or different features which cannot be easily pushed to upstream.

It is my opinion that game theory and technical measures should be considered when designing the upgrade activation. For instance for a certain duration when upgrade adoption reaches over a certain threshold pools that are ready for activation could be rewarded a premium for block production with those who are reluctant to upgrade taking a discount on rewards.

[dnadales]

@nkarag @mciampi ?

[mark-stopka]

Updated data from today:

Version	Nodes	Nodes %	Days since release
0.8.6	3	2.78%	42
0.8.7	1	0.93%	34
0.8.8	1	0.93%	27
0.8.9	62	57.41%	27
0.8.10	0	0.00%	23
0.8.11	16	14.81%	7
0.8.12	25	23.15%	1
Total	108	100.00%

---

Build	Nodes	Nodes %
46df05c7	26	43.33%
30d20d2	8	13.33%
3b1cdb1b	1	1.67%
87c5bc08	1	1.67%
36772bb1	17	28.33%
30d20d2e	7	11.67%
Total	60	100.00%

[mark-stopka]

@michaeljfazio, do you have any comments as to why would someone be motivated to use Jormungandr 0.8.9 other than 46df05c7 which is a latest of your fork releases? Are there any regressions you know of, that would motivate people from upgrading?

[mark-stopka]

@papacarp, would you be able to provide us with the info as to how much stake do these pool running old versions have delegated? Or privately disclose their pool_id so I can fetch the data from https://pooltool.s3-us-west-2.amazonaws.com/8e4d2a3/pools/YOUR_POOL_ID/epochstats.json and add metric showing network stake ownership over time based on Jormungandr version?

[michaeljfazio]

@michaeljfazio, do you have any comments as to why would someone be motivated to use Jormungandr 0.8.9 other than 46df05c7 which is a latest of your fork releases? Are there any regressions you know of, that would motivate people from upgrading?

I think it is simply a matter of pool operators sticking to what's working for them. The reasons why pool operators should update are only going to increase with each release. For instance, there are significant bootstrap improvements in the latest IOHK release that can result in fast and reliable bootstrapping. 0.8.9-alpha-6 does not have these enhancements and therefore rebooting a leader node COULD potentially be problematic some fraction of the time (but definitely slower to bootstrap regardless).

I myself have not yet updated, even though I plan to when "the time is right" for my pool. As it currently stands, I simply have no incentive. At least until I hear from the broader pool operator community that 0.8.12 is stable and efficient. As soon as that happens I'm all aboard.

[dnadales]

BTW @mark-stopka @michaeljfazio I just want to clarify that this repository is related to a research project about decentralized updates for a future version of Cardano. It is not related to Shelley (or the Jormungandr version that runs in the ITN).

Much of the content in this repo is under heavy development, and the first iteration of this research is expected to be finalized at the end of this year.

[dnadales]

I can anticipate however that incentives for updating is not within the scope of our current research. However the practical concern that you @mark-stopka rose is a very valid one and we should see who is going to look into this, and when.

[mark-stopka]

@dnadales, I believe this would be the Voltaire research, correct? I would expect one of the questions that needs to be answered is how big of a risk is it that block-producers would not upgrade to the newer version of the ledger rules, thus blocking fork activation, because I believe in order to prevent excessive hard-forking it's essential that the pools (a.k.a. block producers and validators) that account for 50%+1 lovelace of total delegated stake needs to be delegated to the pools that have upgraded ledger rules.

That would be a bare minimum to justify the safety assumptions of Shelley paper.

I believe data from the current INTv1 for Shelley can be a valuable data to evaluate as to how much of a concern this should be.

[dnadales]

I believe this would be the Voltaire research, correct?

Our current research is not directly related to Voltaire, but funded by https://priviledge-project.eu/. The research output might influence Voltaire, but it's to early to determine this.

I would expect one of the questions that needs to be answered is how big of a risk is it that block-producers would not upgrade to the newer version of the ledger rules, thus blocking fork activation,

Sure. And we should definitely look into this. We just need to find the right moment and scope for it.

I believe data from the current INTv1 for Shelley can be a valuable data to evaluate as to how much of a concern this should be.

Definitely! And it is great that you're reporting this! :pray:

[dnadales]

BTW, I updated the wiki with a link to this issue.