Clarify scoring for season 4 in bible

smk762 commented 4 years ago

Currently scoring is only mentioned in the bible here - Reach at least 33% of your region’s best monthly NN notarization score. This is checked on the first day of each month, beginning one month after the new NN season has begun.

Can we add explicit scoring ruleset? Like

"x" points for a BTC notarisation,
"x" points for a main server notarisation
"x" points for a third party notarisation etc.

Thanks.

Cris-F commented 4 years ago

Currently scoring is:

BTC      1
SC       0.1
3rd      0.2
mined    1

Anyone have suggestions on changes or think this is fine for Season 4?

TheComputerGenie commented 4 years ago

I'd still raise the same objection that, until selection issues are solved, BTC is weighted too heavily that way. Even more so with a tighter race for 3 spots per region and no longer 7-8.

Cris-F commented 4 years ago

Yes, that will become a very big factor this season, until the selection method changes and even after. It does seem that a little bigger incentive for BTC makes sense. Are the goals? Anything else/less?

incentivize notarization participation/competition throughout the entire season
prioritize BTC notarizations (to some extent)

TheComputerGenie commented 4 years ago

incentivize notarization participation/competition throughout the entire season

If you don't get picked as often as others and score low because of that and nothing else, then what is the "incentive"?

Cris-F commented 4 years ago

Yes, I agree. My thinking is to establish the goals of the system and find a solution that meets them. As a very rough example, something like thisBTC 0.4 SC 0.1 3rd 0.2 mined 1We have stats from last year that we (someone other than me) could use to find more appropriate numbers.

Cris-F commented 4 years ago

Mining has its own in-built incentive so that should be easy to adjust. It will also relate to the 'hunting' discussion so that may be able to connect here.

smk762 commented 4 years ago

Perhaps "hunted" blocks (e.g. coinbase > "x") should not get a score point

TheComputerGenie commented 4 years ago

It will also relate to the 'hunting' discussion so that may be able to connect here.

With the min required for that, it's basically moot to points (see #129)

Cris-F commented 4 years ago

In the case of scoring, doesn't 'hunting' basically mean foregoing mining points for (a chance at) $?

smk762 commented 4 years ago

Currently scoring is:
BTC      1
SC       0.1
3rd      0.2
mined    1
Anyone have suggestions on changes or think this is fine for Season 4?

Any chance you have a reference link for this @Cris-F?

TheComputerGenie commented 4 years ago

Any chance you have a reference link for this

navigate to webby's stats site and mouse-over the labels.

TheComputerGenie commented 4 years ago

In the case of scoring, doesn't 'hunting' basically mean foregoing mining points for (a chance at) $?

If #129 is accepted (or the minimum in it is enforced) then the difference will generally be minimal.

ca333 commented 4 years ago

Can we add explicit scoring ruleset? [...]

good call, i propose:

BTC notarisation: 0.5 (1 is indeed too high in consideration of ¹) main server notarisation: 0.25 (50% of BTC notarisation seems like a proportionate reference point) third party notarisation: 0.25 (should be equally valued to AC/main notarisation) mined KMD block: 1

¹ jfyi - in its current implementation the selection of the 13 NNs is indeed "fragile" and predictable over time ref komodo_electednotary() and komodo_chosennotary(). We also noticed performance relation to HW setup and infrastructure which is suboptimal. We are working on dPoW2 and have these points on the radar.

Thanks!

CHMEX commented 4 years ago

I compared the current with the new calculation and it does not make much of a difference just increases the total numbers of scores.

https://docs.google.com/spreadsheets/d/12V6pdrjnIWZaDurrxi5ZzjMhD3jcyJOlAHzM2TrJqC0/edit#gid=0

Results are of course not very accurate because some Operators decided to stop notarizing certain assets after the publication of the auto elected Nodes.

With enough Hash power (doesn't need much) an Operator can increase his scoring by mining SC with the new Scoring system this will have a big impact. This was not much of a problem at the moment because BTC is 10x an SC notarization. With 3P Coins counting the same as an SC NN might not care about those coins. I expect GAME and GIN to drop out so 3P coins will account even less to the total scoring.

I propose we leave it as is. There are fundamental issues with the Notarization Process which are addressed, so lets make it a fairer process for next season.

TheComputerGenie commented 4 years ago

With enough Hash power (doesn't need much) an Operator can increase his scoring by mining SC with the new Scoring system this will have a big impact.

Where did you see something about mining SC chain = points? @CHMEX

CHMEX commented 4 years ago

Sorry for my bad description. I did not mean SC chain = points
but rather the advantage you can/might have if you properly setup the mining node in connection with your NN

Cris-F commented 4 years ago

The spreadsheet is very good @CHMEX, thanks for making it.

smk762 commented 4 years ago

So far we have the following:

option 1 :rocket:

BTC notarisation: 1
main server notarisation: 0.1
third party notarisation: 0.2
mined KMD block: 1

option 2 :heart:

BTC notarisation: 0.5
main server notarisation: 0.25
third party notarisation: 0.25
mined KMD block: 1

option 3 :+1:

BTC notarisation: 0.4
main server notarisation: 0.1
third party notarisation: 0.2
mined KMD block: 1

option 4 :confused:

BTC notarisation: 1
main server notarisation: 1
third party notarisation: 1
mined KMD block: 1

option 5 :eyes:

BTC notarisation: 0.0325
main server notarisation: 0.8698÷numchains
third party notarisation: 0.0977÷numchains
mined block: 0

It's about a week and a half until season change, lets see if we can agree on this before it is last minute.

S4 Notary operators: please add an emoji matching your favored option to this post to indicate your preference.
If you want to add an alternative option, respond with details and an indicative emoji to count votes towards it.
If you dont like any proposal, but have no alternative to propose, respond with :fishing_pole_and_fish:
if you dont care either way, respond with :bath:
voting for more than one option (i.e. adding more than one emoji) will split your vote proportionally over the options indicated.

:beers:

TheComputerGenie commented 4 years ago

Which emoji is for "none of these make sense"? Option 4: BTC notarisation: 1 main server notarisation: 1 third party notarisation: 1 mined KMD block: 1

Also, which emoji is for "Ive got better things to do than google how to \<insert emoji here>"?

smk762 commented 4 years ago

Which emoji is for "none of these make sense"? Option 4: BTC notarisation: 1 main server notarisation: 1 third party notarisation: 1 mined KMD block: 1

Also, which emoji is for "Ive got better things to do than google how to "?

maybe the : confused : emoji ? Here's a link for convenience - https://gist.github.com/rxaviers/7360908 I tried to start with the standard emojis from next to the three dots in the top right.

greerso commented 4 years ago

Do mined blocks need a score, isn't the block reward score enough? If anything negative score if the ratio of mined blocks to notarizations is lower than the average across all other notaries.

NOCTLJRNE commented 4 years ago

I kinda agree with CG, if dPoW 2.0 can't be done in time to be ready for ss4, then we should BTC nota weight equally with the rest.

tonymorony commented 4 years ago

if dPoW 2.0 can't be done in time to be ready for ss4, then we should BTC nota weight equally with the rest.

dpow 2.0 isn't something what can be delivered until the next season

gcharang commented 4 years ago

Here is my attempt at simplifying the numbers to make the impact of the ratios on different scoring systems easier to digest.

All the options give mining a KMD block 1 point. As it is in a notary's financial interest to do so and afaik, the process isn't random and the requirement is very simple, we can assume most notaries would mine every block they are allowed by the consensus. So, we can remove this metric's impact on the scores as most notaries should have a similar number of total mined blocks over a whole season.

Now, to convert the scores of each option to an easily digestible format, I am representing them as ratios that result in a Hundred when each component is added together. This way, options can be compared with each other and reasoned about easier. If this was a school test, each number below represents the percentage of score reserved for the corresponding item.

	BTC nota	main server nota	third party nota
Option 1	76.92	7.69	15.38
Option 2	50	25	25
Option 3	57.14	14.28	28.57
Option 4	33.33	33.33	33.33

I have double-checked the numbers, but can someone else please verify and leave a comment if they do accurately represent the above-proposed options?

Cris-F commented 4 years ago

I'll go with 3.
More incentive for BTC and 3p should be useful for other areas of the project like Business Dev and Marketing, but not as much as S3. Also provides some incentive to improve fairness while waiting for DP 2.0. Ways of reducing cost of tiny BTC splits (eliminate them?) for example.

ca333 commented 4 years ago

I kinda agree with CG, if dPoW 2.0 can't be done in time to be ready for ss4, then we should BTC nota weight equally with the rest.

the thing with 3P/main notarisations is that they are "lot cheaper"

So far we have the following:

option 1 🚀
BTC notarisation: 1
main server notarisation: 0.1
third party notarisation: 0.2
mined KMD block: 1
option 2 ❤️
BTC notarisation: 0.5
main server notarisation: 0.25
third party notarisation: 0.25
mined KMD block: 1
option 3 👍
BTC notarisation: 0.4
main server notarisation: 0.1
third party notarisation: 0.2
mined KMD block: 1
option 4 😕
BTC notarisation: 1
main server notarisation: 1
third party notarisation: 1
mined KMD block: 1
It's about a week and a half until season change, lets see if we can agree on this before it is last minute.

S4 Notary operators: please add an emoji matching your favored option to this post to indicate your preference.

If you want to add an alternative option, respond with details and an indicative emoji to count votes towards it.

If you dont like any proposal, but have no alternative to propose, respond with 🎣

if you dont care either way, respond with 🛀

voting for more than one option (i.e. adding more than one emoji) will split your vote proportionally over the options indicated.

🍻

@smk762 can you plz start a poll with these inside the NN group (or refer them here) and based on the result we just re-hash inside the team and make a final decision.

my two komodoshis: Scoring all notas equally is likely suboptimal as the BTC notas are higher in (tech) "value" and main/3P notas can be easily "scaled up" (= the nota numbers are not within a constant scope like BTC notas). Yes, the mining score could be removed ref. the financial incentive. I personally believe setting 3p and main score equal and BTC score 2x (or even 4x) that value should give a good and motivating ratio.

daemonfox commented 4 years ago

I put a bunch of detail on my thought process into the #priv-notarynode discord channel. The below is scaled based on S3 AVG calculated EXPECTED NN Op performance, if all 64 nodes were equal. This would yield a comparable amount of points per type and awards those who keep all 4 running at or above AVG expected rate each day.

TYPE	Pts Per Type	AVG Pts Per Day
BTC	0.5	14.25
SC	0.02	16.19
3P	0.20	17.09
BM	1.00	15.83

TheComputerGenie commented 4 years ago

In amount of "work"*

work percentages
KMD->BTC=    3.25%
ACs->KMD=   86.98%
3P->KMD=     9.77%

*of all stats on webby's site main page (I'm too lazy to dig for the hidden bits and/or do a full chain scan)

IMO, anything that isn't "the same" across the board should reflect that rather than something arbitrary or subjective and pulled out of the air.

TheComputerGenie commented 4 years ago

TYPE	Pts Per Type
BTC	0.0325
SC	(0.8698÷numchains)
3P	(0.0977÷numchains)

ca333 commented 4 years ago

So far we have the following:

option 1 🚀
BTC notarisation: 1
main server notarisation: 0.1
third party notarisation: 0.2
mined KMD block: 1
option 2 ❤️
BTC notarisation: 0.5
main server notarisation: 0.25
third party notarisation: 0.25
mined KMD block: 1
option 3 👍
BTC notarisation: 0.4
main server notarisation: 0.1
third party notarisation: 0.2
mined KMD block: 1
option 4 😕
BTC notarisation: 1
main server notarisation: 1
third party notarisation: 1
mined KMD block: 1
option 5 👀
BTC notarisation: 0.0325
main server notarisation: 0.8698÷numchains
third party notarisation: 0.0977÷numchains
mined block: 0
It's about a week and a half until season change, lets see if we can agree on this before it is last minute.

S4 Notary operators: please add an emoji matching your favored option to this post to indicate your preference.

If you want to add an alternative option, respond with details and an indicative emoji to count votes towards it.

If you dont like any proposal, but have no alternative to propose, respond with 🎣

if you dont care either way, respond with 🛀

voting for more than one option (i.e. adding more than one emoji) will split your vote proportionally over the options indicated.

🍻

@last bump - we will calc the result this week and merge it into the bible before S4 kicks in.

smk762 commented 4 years ago

As votes are currently tied, tiebreaker system is as below -

In the event of a 2 way tie, the last char of the blockhash for block 1922000 will act as a tie-breaker (even digit or b/d/f == vote for lowest option number, odd digit or a/c/e == vote for highest option number)
In the event of a 3 way tie, I'll use last char tie-breaker for blocks 1922000-1922002 as above, with lowest option vs middle option, middle vs high, and low vs high in that order, repeating for consecutive 3 block ranges as required until tie is broken.

TheComputerGenie commented 4 years ago

Looks untied to me

smk762 commented 4 years ago

pending multinode votes

TheComputerGenie commented 4 years ago

Or they did until someone threw a fit and in priv chans @smk762 says

I'll contact with multi-node operators to get their votes, and see what the complete tally is. If it ends up with option 5 leading, I'll start a poll to repeal and replace it.

No clue why you wasted everyone's time

smk762 commented 4 years ago

You're free to start your own poll. This issue was initially intended to include details on scoring in the nn bible, it evolved into a poll as I was trying to be inclusive. The reasons for option 5 being difficult to implement have been discussed, and the question of multinode votes were also raised. There is nothing wrong with follow up votes, any notary can submit them - tho the abuse I've copped for trying to be inclusive even before the vote ended is sad. No doubt processes could be improved - github sucks for counting polls, and this whole effort was a waste of my time also. I'd likely have been abused less for submitting a PR rather than asking for input.

To summarise the reasons -

no mining score component risks chain stall and incentivises undermining (i.e. hunting) effectively rewarding with score those that do the least work in terms of nn operator duty iii from the notary bible.
"numchains" was not sufficiently defined, and there are challenges with maintaining this if/when new chains are added or removed.
the code for tallying scores for dashboard display becomes more difficult to maintain, and those responsible for maintaining that code consider it more labour intensive than is ideal.

For the reasons above, I consider it prudent (in the event of option 5 being the winner after multinode operators confirm) to undertake another poll which could alleviate some of these concerns.

This issue was to "clarify", not add complexity, and without an explicit definition of "numchains", clarity is suboptimal.

Once multinode ops have indicated their additional votes, if option5 wins, it stands until another poll defeats it. You can add your options to any subsequent polls (though I would ask that explicit definitions are included, for clarity).

There is nothing to compel third parties to display scoring on their dashboards if they dont see the effort taken to implement as worth it.

smk762 commented 4 years ago

multinode vote counts moved to here - https://docs.google.com/spreadsheets/d/1jLqpbHUm0EweMFGnTAaEgXXJI4zj-cVfLCxFaMHjBDM/edit?usp=sharing

TheComputerGenie commented 4 years ago

not accurate, but should serve as reference enough for conversation CREATE TABLE webby_lazy (NAMEvarchar(9),CHAIN_TYPEvarchar(2)) ; INSERT INTO webby_lazy (NAME,CHAIN_TYPE) VALUES ('NINJA', 'AC'), ('SUPERNET', 'AC'), ('AYA', '3P'), ('DEX', 'AC') ; INSERT INTO webby_lazy (AC_COUNT) VALUES (SELECT CHAIN_TYPE, count() as 'numchains' FROM webby_lazy WHERE CHAIN_TYPE='AC'); INSERT INTO webby_lazy (3P_COUNT) VALUES (SELECT CHAIN_TYPE, count() as 'numchains' FROM webby_lazy WHERE CHAIN_TYPE='3P');

CHAIN_TYPE	numchains
AC	3

CHAIN_TYPE	numchains
3P	1

TheComputerGenie commented 4 years ago

and without an explicit definition of "numchains", clarity is suboptimal.

eg above, "explicit definition" is math

smk762 commented 4 years ago

or you can parse from here - http://notary.earth:8762/api/info/coins/?dpow_active=1

There is still potential for chain removal/addition/server change to disrupt the denominator tho unless we have explicit start/end blocks defined when a change like this occurs as mentioned in https://github.com/KomodoPlatform/dPoW/issues/120

I still believe it is more complex than is ideal in terms of labor overhead from the perspective of someone working on a stats dash (and that the work of mining should be proportionately rewarded by score), but I'm sure that's been mentioned already.

TheComputerGenie commented 4 years ago

It doesn't matter where a coin moves to. Score is the sum of all days thus far (score now = yesterday + today) and an additional column or 3 isn't "complex" but rather the entire point of a database. "Labor overhead" of adding a few columns to a db is less than it took for you to type your last reply @smk762

daemonfox commented 4 years ago

I believe what you just typed is REQUIRED by the very nature of what we are doing here.

"...to disrupt the denominator tho unless we have explicit start/end blocks defined when a change like this occurs as mentioned in #120 https://github.com/KomodoPlatform/dPoW/issues/120" Is something I would expect to be already tracked and notated as it pertains to the life of that projects notarisation. By default this data is available (activation ht and time stamp of notas) so it should be a small effort to add it to a stats tracker calculation. No need to worry about filtering extra data either... as a coin enters and leaves, they should only have notas within that time frame, so scoring will always stay bucketed within start and final notarisation.

On Mon, Jun 15, 2020 at 10:32 AM smk762 notifications@github.com wrote:

or you can parse from here - http://notary.earth:8762/api/info/coins/?dpow_active=1

There is still potential for chain removal/addition/server change to disrupt the denominator tho unless we have explicit start/end blocks defined when a change like this occurs as mentioned in #120 https://github.com/KomodoPlatform/dPoW/issues/120

I still believe it is more complex than is ideal in terms of labor overhead from the perspective of someone working on a stats dash (and that the work of mining should be proportionately rewarded by score), but I'm sure that's been mentioned already.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/KomodoPlatform/dPoW/issues/135#issuecomment-644171746, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABGPIAPKUYMB6XWPCXTHOGLRWYWHJANCNFSM4NIWXGBA .

TheComputerGenie commented 4 years ago

Since I'm not even allowed in the community any more, I'll leave you with this:

If Komodo is boiled down to what takes the least amount of work to do because random and arbitrary is easier than something thoughtful that is based on historical data (± whatever wasn't available at the time because making things disappear is easier than actually calculating them), then we've all just wasted our collective years and it's doomed...

smk762 commented 4 years ago

Both you and web copped a timeout from discord for abusive conversation, after being warned and reminded of Accessing the private NN Discord community chat, as long as a mature and respectful basis of discussion is given. Wasn't my call, but follow up conversation has been much less stressful.

Web has worked on his stats for years, and is within his rights to do what he wants with it - it was a community service. He has unique insight that comes from this experience,

I dont have a problem with the idea behind relating scoring to work done, as long as it includes the task of mining - I've been pretty clear about that, its the basis of my opposing opinion. The ambiguity of "numchains" and contingencies related to changes needed more thought, which seems to have been provided here, and could have been done in discord without resorting to insult.

It's difficult and time consuming to operate in such a toxic environment, it drives away contributors and results in some deciding not to engage in the discussion which is counterproductive. Allowing such an environment to persist would surely doom future efforts.

Alternatively, if we can disagree without being subject to abuse, maybe we can communicate our concerns and views effectively and negotiate consensus without extended periods of distress along the way.

On the upside, the failings of this poll and the process of attempting it has offered some insight into the flaws exposed and how to improve it in future, like setting up well defined options before the poll even begins, what participation rate would be sufficient for ensuring consenus, using a better platform for voting etc. so future polls are less prone to being derailed. It's an iterative process, but it wont evolve without those participating showing respect for the process and those involved.

TheComputerGenie commented 4 years ago

I wanted to go away, but

The ambiguity of "numchains"

How is there "ambiguity" in a literal number? If there are 3 chains then numchains =3, if there are 30 chains then numchains =30, and if there are 300 chains then numchains =300.

the failings of this poll

...seem to be nothing more than you don't like the outcome and will do everything you can imagine to force a different outcome.

so future polls are less prone to being derailed

So, in the future you'll accept the opinions of other but just not now.

showing respect for the process and those involved

You mean like Strob voting for whatever will "defeat option 5" and not caring what the actual result is as long as it's "anti-CG"? There's no "respect" in any of this process, if there were then no one would have been banned today and "voting" wouldn't be done in secret DMs, manually written on a google spreadsheet by you, and where folks just take your word for it.

StrobFX commented 4 years ago

To defeat it not because it's from you CG, but because it doesn't take voting into account. I don't like that idea that's all. That said I'm against how you communicate globally.

smk762 commented 4 years ago

The only "secret" vote done by DM was one person, who abstained. Others posted in discord, and the link to their post is in the spreadsheet as a comment in an adjacent cell.

greerso commented 4 years ago

Do mined blocks need a score, isn't the block reward score enough? If anything negative score if the ratio of mined blocks to notarizations is lower than the average across all other notaries.

Adding to my previous question about points on mining. I now understand and support the need for it.

TheComputerGenie commented 4 years ago

Given that there's a minimum requirement of 15 blocks a day, all mining points are is a "participation trophy". 5500 meaningless points at one per block or whatever the factor of that would be. "No points will encourage hunting" ...said no one that's ever had to pay for the hashrate to get 15 KMD blocks per day every day for an entire season.

KomodoPlatform / dPoW

Clarify scoring for season 4 in bible #135