Document that a decreasing value in NIL theta is good

[friedger]

What is the problem you are seeing? Please describe. The NIL theta decreases with the number of good applications.

How is this problem misaligned with goals of app mining? It looks like that it is unfair to get lower NIL score for apps with full points with the increase of good apps. However, the negative effect for apps that do not have full score increases more.

The image below shows a simulation for 10, 100, 200 apps where on app has score 0, and the rest score 4. The first row is for the bad app, the second row for the good apps.

What is the explicit recommendation you’re looking to propose? Add a paragraph in the documentation that explains this effect and the effect that NIL becomes less relevant for the good apps, but very relevant for the few bad apps.

Describe your long term considerations in proposing this change. Please include the ways you can predict this recommendation could go wrong and possible ways mitigate.

Additional context

Spreadsheet with simulation https://app-center.openintents.org/appco/924

[hstove]

Add a paragraph in the documentation that explains this effect and the effect that NIL becomes less relevant for the good apps, but very relevant for the few bad apps.

I would not be opposed to updating our docs, but this is more around a "practical" result than a design goal. The way NIL works currently is that it really forces you to get 100% on digital rights, which I think is great. It does not mean it's really "irrelevant", it just forces a certain design of your app.

[friedger]

@hstove Yes, indeed, that paragraph would only be for educating app publishers with full NIL score to clarify that it is good for the ecosystem if their NIL theta is dropping

[Walterion01]

As I said in #7, NIL rank seems irrelevant. If there is no practical difference on Gaia usage, just requiring Blockstack Auth that already is needed on submission, seems enough. NIL has good potential to make a beneficial impact on overall results, but in this approach, it is not very effective.

[friedger]

@Walterion1 This issue is more about the fact that if all apps receive full score (theta = 0 for all) is good. Because then one new app that does not receive full score will have theta -10.. or so. This is not about whether the score represents digital rights well enough.

[hstove]

I don't think we should add documentation here - the average NIL score could go up or down month to month. This is not a built-in mechanism, it's just the way apps are behaving.

[friedger]

Educating developers should help the ecosystem :-)

The score could go up and down, however, it only depends on the majority of apps not on the app itself if the app is unchanged. For other reviewers, it is less apparent because the range of values is much bigger and it is much more difficult for the majority of apps to achieve the maximum score for the other reviewers.

It is a built-in mechanism that the theta for an app (that received the maximum score) goes down faster towards 0 the more apps receive the maximum score.

[moxiegirl]

An FAQ might help "How does NIL work" or some such.

[Walterion01]

@moxiegirl the post blog about NIL is not describing the exact situation right now so a check with @larrysalibra would be useful. More info at #7.

[stackatron]

Discussed with PBC team, how NIL works is already in our docs. And this topic is covered by the results key. Closing this.