Digital Rights reviewer: apps listed as 'broken'

[hstove]

I'm opening this to start a discussion, and I'm not personally announcing any hard decisions in this ticket.

What is the problem you are seeing? Please describe. Some apps were listed as 'broken' in the dry run for the new reviewer. From my quick check, they were working just fine.

How is this problem misaligned with goals of app mining? We need to be consistent and transparent before strongly punishing apps.

What is the explicit recommendation you’re looking to propose? I think the new reviewer should publish what environment they're testing on - OS, browser, etc. Then, developers could at least be sure to test against that environment.

[thedavidlewis]

5 dapps received a -1 for Auth. This was easy to verify.

3 should have received a 4 (Blockstack required) 2 should have received a 2 (Blockstack as one of several but prominently displayed)

I've read in other posts that some dapps don't have Blockstack Auth implemented... it seems they got a 4.

It's a good thing this was a dry run for March!

[larrysalibra]

Here's an example of an app, Springrole, where sign in with Blockstack was broken:

springrole.mov.zip

In this particular case, it appears their oauth endpoint is rejecting the authentication response from my sign in request.

---

I think the new reviewer should publish what environment they're testing on - OS, browser, etc. Then, developers could at least be sure to test against that environment.

In the wild, users use a heterogenous mix of environments. Making sure your app works in these different environments is basic requirement for shipping a product for the web.

If your app is broken in the user's environment, you lose the customer.

We'll use a common configuration to test. We want great apps that work well, not apps that are "developed to pass the test".

I'm happy to share test environment details after the test to make it easier for app developers to locate and repair the problems:

This time around we used macOS 10.14.3 (18D109), Safari Version 12.0.3 (14606.4.5) and Blockstack for macOS v0.35.4 (116) to test.

---

We need to be consistent and transparent before strongly punishing apps.

App Mining is intended to reward apps with characteristics that it incentivizes for. Using the word "punishing" implies that there's some sort of entitlement to receive App Mining rewards.

---

From my quick check, they were working just fine.

That sounds like the "Works on my Machine" certification problem: https://blog.codinghorror.com/the-works-on-my-machine-certification-program/

If I'm a potential user, I don't care if it worked for you or for the app developer. If it didn't work for me, I'm going to bounce and use something else.

[thedavidlewis]

@larrysalibra Thanks for the quick response.

[blockcred]

Hello @larrysalibra

According to "Blockcred"

In the dry-run review, we had the following score: "App Citzen Gaia"= 2 .

As per the scoring methodology supported on the following link:

https://blog.blockstack.org/introducing-new-internet-labs-the-digital-rights-reviewer-for-app-mining/

Score = 2 means, (Gaia + others – Data is stored to Gaia and also another service (possibly an indexer controlled by the app developer)).

which is not correct for "Blockcred".

If the check performed before our latest update, the score should be ZERO, because at that time we didn't use GAIA.

If the check performed after our latest update, the score should be FOUR as we became fully utilizing GAIA through entire the whole App and don't use any other DB or indexing tools.

So, score "2" is not correct. and affected the average score negatively, which also affected "App Citzen Score" and "App Citzen Theta" accordingly

The score should be either ZERO nor FOUR depends on the time of reviewing.

Thanks

[larrysalibra]

Hi @blockcred

When I performed the test, there was a POST request to https://app.blockcred.io/api/radiks/models/crawl that failed with an HTTP error code.

I just tried again and the request to that URL succeeded this time. The URL appears to be an indexing server. It's certainly not my Gaia hub. Based on that, 2 is correct.

[blockcred]

Thanx @larrysalibra for your fast response, I really appreciate it.

It is returned from Radiks server.

based on your last check, 2 still correct?

[larrysalibra]

based on your last check, 2 still correct?

Yes. We rated apps that only store data in gaia hubs as 4 and those that store in gaia hubs and some other developer or 3rd party controlled server (including Radiks) as a 2.

When we start doing this for real next month, we're thinking of changing the rating system so that there's no difference between the two. In that case, using gaia might get you a 4 and there would be no 2 rating.

As a bit of background: in an ideal world, apps would store all user data in users' gaia hubs and operate without any other servers. In this way, users' wouldn't be locked in or reliant on the continued existence of a particular application company.

While that's possible with certain categories of apps today, it's very challenging - perhaps not possible. For other categories of apps.

The question we have to decide as a community is what digital rights tradeoffs are appropriate in the name of additional functionality.

I'd love to discuss this more on the upcoming App Mining call if you have time to join!

[brycedev]

In the wild, users use a heterogenous mix of environments. Making sure your app works in these different environments is basic requirement for shipping a product for the web.

If your app is broken in the user's environment, you lose the customer.

We'll use a common configuration to test. We want great apps that work well, not apps that are "developed to pass the test".

I'm happy to share test environment details after the test to make it easier for app developers to locate and repair the problems:

This time around we used macOS 10.14.3 (18D109), Safari Version 12.0.3 (14606.4.5) and Blockstack for macOS v0.35.4 (116) to test.

App Mining is intended to reward apps with characteristics that it incentivizes for. Using the word "punishing" implies that there's some sort of entitlement to receive App Mining rewards.

From my quick check, they were working just fine.

If I'm a potential user, I don't care if it worked for you or for the app developer. If it didn't work for me, I'm going to bounce and use something else.

If an application is confirmed to be working by both users, developers, and TryMyUI users, but not for you, Larry, this is an indication that there may be a timing bug, or an issue with your machine. We can't assume it simply doesn't work, when there are indications otherwise. How can the statements here be reiterated by several participants, and you shrug them off as user error. A lot of of us have built products for a mix of environments before. We are dealing with experimental software run by Blockstack and distribution; be patient with us, and not jump to conclusions. At least ask us what our software uses, or if we are currently doing testing, or having issues, perhaps?

"If I'm the potential user." You aren't the potential user. You are the Digital Rights reviewer, right?

Additionally, why are there such high standards, now? Now, we must work perfectly. Prior to this, and currently, developers are rewarded for doing just this, hitting a precise metric for this program : working on Larry's machine, getting DE votes from being in an early cohort or having an affiliation, getting a certain amount of upvotes. This is the reason for the angst, and the need for you to act as the best possible reviewer, free from yourself and ego. This isn't about "I, the potential user". This is about Digital Rights being upheld. You must investigate how they are being upheld at any cost. That's "your" assigned duty.

[hstove]

Hey @blockcred , please continue discussion around using Radiks to #58, so we can keep this issue on target.

[jyudkin1]

@larrysalibra was wondering what was broken or the error that caused the scoring of -1 in all categories? Forms.id is 100% distributed/p2p from the beginning. Blockstack Gaia, IPFS, and LibP2P.

Do things that don't scale is probably the best thing here - simple solution is to maybe create a form (feel free to use forms.id ;)) to have participants share specs, code review / github, schedule a call, not really sure why this isn't something that is manageable.

Lets say there are 100 apps in the program, you can easily "review" every app - I know thats not the goal, but it seems like there is a lot of human involvement with a huge opportunity to engage the mining community and be the best reviewer.

I also think that reviewers have a massive responsibility here in the early days, disproportionately so - again, a slack message or a quick email or something could have solved this type of error in scoring.

[thedavidlewis]

@larrysalibra In the video you provided for SpringRole, there is a bug that in certain usecases (I can recreate it on my Macbook but no one else on the team can) the Blockstack login is in the original tab and the original page is in the newly-spawned tab. We're looking into this to see if it is on our end or Blockstack. Two questions:

1) Will there be a process for us to have you verify you see it working prior to the next month's testing? 2) Did you give us a -1 for Gaia due to this?

[dantrevino]

Should we really be dinging apps for not supporting a browser with 5% market share? What if it works on Safari, but doesn't work on Firefox? I can see two sides to this but wanna raise the question.

[friedger]

Digital Rights Reviewer should not test usability, performance, correctness. If it fails during test I propose to just say Ineligible.

See https://github.com/blockstack/app-mining/issues/35#issuecomment-469450497 the solution to https://github.com/blockstack/app-mining/issues/51

[friedger]

This is also related to #66 where a communication channel is requested between reviewers and app publishers. and to #63 where a clone of vote.blockstack.org is suggested, which can be used as communication channel.

[jyudkin1]

Digital rights reviewer should at least share what their testing environment is - what if they are using IE?

Again, I think the goal isn't to optimize cross browser compatibility here - this test is to evaluate Digital Rights and how an end user would experience it.

If an end user can't login or authenticate in IE, it really doesn't matter - unless we are scoring apps on browser compatibility. That seems like a different test / outcome with this reviewer if that is the case - I believe this is not the goal.

[geeogi]

Hi @larrysalibra

We at Zinc are keen to improve our understanding and implementation of Blockstack with respect to the user’s digital rights.

We’ve integrated Blockstack login as a recommended login option yet we scored -1 on this month’s dry run. How can we find the reason for this score?

[larrysalibra]

@geeogi

We at Zinc are keen to improve our understanding and implementation of Blockstack with respect to the user’s digital rights.

Great!

We’ve integrated Blockstack login as a recommended login option yet we scored -1 on this month’s dry run. How can we find the reason for this score?

Whe we tried it, we received an error after being redirected back to Zinc's website:

400 { "message": "{\n \"Error Message\": \"Invalid params. Does not meet any of the handled login methods\"\n}", "statusCode": 700 }

I just tried it again and it looks like it's still happening.

Attached is a video

zinc.mov.zip

My first guess would be that the authentication endpoint is configured to only accept redirects from https://browser.blockstack.org - authentication requests can come from any URL. In the case of the default configuration of the Blockstack mac/windows/linux installation, redirects come from http://localhost:8888.

Hope that helps!

[larrysalibra]

was wondering what was broken or the error that caused the scoring of -1 in all categories?

When I signed in, the app stopped at the translucent blue overlay with spinner.

I just tried to sign in again - since i'm still signed in, it returned to that state.

The error appears to be a null pointer on App.vue line 60. I'm not sure if that's the original error that resulted in this state though.

You can see the error in the console at the end of this video:

forms.id-720p.mov.zip

I cleared localstorage again and tried logging in again and everything seems to work. Looks like the bug that resulted in the null pointer state is fixed!

Forms.id is 100% distributed/p2p from the beginning. Blockstack Gaia, IPFS, and LibP2P.

Cool! I look forward to digging into your use of IPFS and LibP2P as well. Very exciting!

[larrysalibra]

@friedger

Digital Rights Reviewer should not test usability, performance, correctness. If it fails during test I propose to just say Ineligible.

Our goal is to reward App developers that make great apps that use both Auth and Gaia. These apps give users control over their identities and their data - a huge move in the direction of improved digital rights. The vast majority of apps implemented auth and/or gaia without any issues. We want to reward...incentivize more behavior like those developers.

To say "you added a Sign in with Blockstack ID button but it didn't work so we'll just not count the results of the digital rights review" is unfair to app developers that got things right and incentivizes the wrong kind of behavior.

[larrysalibra]

@jyudkin1

Digital rights reviewer should at least share what their testing environment is - what if they are using IE?

I think it makes sense to set rough guidelines for testing environment. Something like "review will be conducted on the current public release of mac, windows, ios or android with either the default browser on the platform or the current release of chrome or firefox."

It's better for users' digital rights if they can use whatever browser they normally use. This makes me disinclined to announce a specific platform/browser combination in advance. In this dry run, we reviewed with macOS and Safari. Next month might be Windows 10 and Firefox. Or it could be Android and Chrome.

Thoughts?

[larrysalibra]

Do things that don't scale is probably the best thing here - simple solution is to maybe create a form (feel free to use forms.id ;)) to have participants share specs, code review / github, schedule a call, not really sure why this isn't something that is manageable.

Lets say there are 100 apps in the program, you can easily "review" every app - I know thats not the goal, but it seems like there is a lot of human involvement with a huge opportunity to engage the mining community and be the best reviewer.

These are excellent suggestions. We'll discuss with the Blockstack PBC team this coming week and see what we can come up with.

[jyudkin1]

@larrysalibra it becomes a browser compatibility test, I don't think that is the intention here.

Digital rights are only at risk when the app functions as intended.

In order to encourage rapid development / improvement / focus - I believe most early stage product folks would agree, focus on one platform, and one browser (Chrome due to adoption). Same goes for Mobile vs Web, iOS vs Android, etc.

Limitless amount of edge cases in one browser already ie Chrome. I think for highest impact / highest chances of market fit, developers should triage these browser opportunities based on impact and scale in order to find leverage where they can.

Established Tech co's and products can't solve this either - cross browser compatibility is an ongoing problem, seems like a very high bar for mining program (this early).

All of this is compounded with building on Blockstack ( since it is still early days ;)).

Something to consider - If browser compatibility becomes a function in mining program, it will encourage slower production times for innovations and/or simple static apps / gaia wrappers / less complexity.

Building is generally a prioritization exercise.

[friedger]

Our goal is to reward App developers that make great apps that use both Auth and Gaia... To say "you added a Sign in with Blockstack ID button but it didn't work so we'll just not count the results of the digital rights review" is unfair to app developers that got things right and incentivizes the wrong kind of behavior.

There is a difference between a broken login with your setup and a non-functional button. If the app is open-source (see #11) than that is easy to distinguish. But also trying different setups might help (see below).

It's better for users' digital rights if they can use whatever browser they normally use. This makes me disinclined to announce a specific platform/browser combination in advance. In this dry run, we reviewed with macOS and Safari. Next month might be Windows 10 and Firefox. Or it could be Android and Chrome.

I like that you are changing the testing setup. However, if the feature is broken on your setup of the month, you should change it for the failing app and use the last known setup to work or try 2 or 3 other setups. If you find that it is indeed a problem with a particular setup you could reduce the final score by one or two points depending on the setups you needed to try before getting it to work. That way you would also value an app higher that respects all digital rights, but does not work on your current setup than an app that only use auth but across all possible setups.

[geeogi]

400 { "message": "{\n \"Error Message\": \"Invalid params. Does not meet any of the handled login methods\"\n}", "statusCode": 700 }

Hi @larrysalibra thanks for that. That's very helpful. We're pushing out a fix today for this issue.

[stackatron]

Spoke with @larrysalibra about this topic. Please keep in mind that the vast majority of apps did not have this problem. And that our ultimate goal is to promote app quality, which is challenging when bugs block users from even trying certain apps. Our proposal:

• Digital rights reviewer will not provide an exact testing environment, but will test various environments changing month to month, with the following guidance:

review will be conducted on the current public release of mac, windows, ios or android with either the default browser on the platform or the current release of chrome or firefox.

• If your app fails this test, by definition you're not eligible for App Mining since your app does not have working Blockstack auth.

• For April App Mining, the Digital Rights reviewer will test all apps, then notify all the apps that fail via email. Then 48 hours later, the failing apps will be retested. Only then will they be disqualified for the April cohort.

We can try this solution then reevaluate. Our goal is to remove the 48 hour notification period. We're including it now to be fair and helpful while introducing this change.

Moving this ticket to QA. Feedback welcome. Thanks.

[jyudkin1]

@jeffdomke Seems like this doesnt really solve the larger issue. Lets call this test a browser compatibility test then.

Issues with this - will the chrome / Mozilla etc setups be extension free? Antivirus free ? etc etc. How bare bones will the environment be? Ad blocker free etc. Those are all variables that would affect the test.

Lets call it how it is here. This is a browser compatibility test.

Again - If we are going to optimize any browser or machine setup, why dont we look at the webs distribution of browsers? Chrome vs IE vs Mozilla vs Obera vs etc

The issue here is prioritization and really thinking about impact and what needs to be optimized: http://gs.statcounter.com/browser-market-share/desktop/worldwide

Safari @ 5% desktop Firefox @ 8% desktop Chrome @ 71% desktop

If you are a startup or early stage company fighting to build and scale something - which of these browsers would you optimize for? Which edgecases / which bugs would you build for?

@larrysalibra would love to chat about this.

[thedavidlewis]

Based on what we’ve found, I’m guessing @larrysalibra is testing with the Blockstack app installed. I have it installed and no one else on our team does. The Blockstack app spawns an additional browser tab. It shouldn't. Worse is that for some apps it spawns the tab with the dapp's login screen (or an error) and displays the Blockstack browser page in the covered tab.

We have a fix in our next release. My concern is that new dapps will not be eligible if they this bug affects them and they'll have to waste resources trying to fix it.

(@larrysalibra and @hstove: Can you test logging in at springrole.com with and without the Blockstack app installed to test the theory? Thanks!)

[Pierre-Gilles]

Hey @jeffdomke!

I noticed Gladys received a score of 0 on Digital Reviewers Rights, despite Gladys-Blockstack integration using both Auth + Gaia storage.

As I understood, this is an automated test, and by the nature of Gladys (it's a self-hosted product), I can understand that this automated test doesn't work in this case (it has probably scanned Gladys commercial website and not the product itself).

Do you see any way we can handle this kind of product? How did you do for mobile apps?

[sdeepak23]

Hi @larrysalibra, We just noticed that Pden has been given a 0 for digital rights Gaia score even though we use it for storing our user posts. Can you share what the issue was?

[GinaAbrams]

Tagging @larrysalibra to comment here. Thanks! 🙏

[larrysalibra]

We just noticed that Pden has been given a 0 for digital rights Gaia score even though we use it for storing our user posts. Can you share what the issue was?

In the dry run, we were unable to verify whether or not mobile apps were using gaia. In upcoming april round, we plan to use new methods to check this. If you're using gaia then you shouldn't have a problem in the next round.

[larrysalibra]

@Pierre-Gilles

I noticed Gladys received a score of 0 on Digital Reviewers Rights, despite Gladys-Blockstack integration using both Auth + Gaia storage.

We were unable to test Gladys (which looks really cool by the way) because we don't have that hardware that Gladys runs with or the hardware that it controls. It also appears that even given access to the the hardware, it would take a time investment on the order of hours to get everything up and running.

How did you do for mobile apps?

We have access to Android and iOS mobile phones.

We have android and iOS devices to test on.

As I understood, this is an automated test, and by the nature of Gladys (it's a self-hosted product), I can understand that this automated test doesn't work in this case (it has probably scanned Gladys commercial website and not the product itself).

It's a human-controlled manual review where we examine the behavior of the app. We let the Blockstack PBC team know that we weren't able to test Gladys and other apps that require hardware beyond the major mobile and desktop platforms.

I think it would be great if we could figure out a way to include apps like Gladys that make novel uses of Blockstack's technology. One idea would be source code review, but it's not always trivial to know how code behaves when used especially for someone not familiar with a system.

How do the other app mining reviewers handle apps such as this? cc @GinaAbrams

[stackatron]

Sounds like we are moving forward Larry's plan for this month.

[fiatexodus]

In the dry run, we were unable to verify whether or not mobile apps were using gaia. In upcoming april round, we plan to use new methods to check this. If you're using gaia then you shouldn't have a problem in the next round.

@larrysalibra if this proves problematic, let me know as I can help you determine / see how Stealthy is using GAIA (GAIA is used to store contacts, conversations, outgoing messages etc.--pretty much everything). It's easy to see some of it by just looking things up on the core end point, but I imagine you might want a dynamic check to see something you do in app change a file on GAIA directly.