raid-guild / indexed

A collectively built hub for comparison between "Layer 2" solutions
8 stars 27 forks source link

Errors / inconsistencies on the latest live site #138

Closed DZGoldman closed 3 years ago

DZGoldman commented 3 years ago

I saw index.wtf was back online and I’m still seeing a lot of inaccuracies / inconsistencies; I had submitted info via the form for Arbitrum (a project I’m directly involved with) a few weeks back, but it seems to have not been incorporated. The info I submitted is here:

Tldr: Arbitrum is an Optimistic Rollup protocol, not POS/Plasma, is permissionless, and has the same security guarantees/finality as the L1. Much of the info on the “/arbitrum-doc” tab appears to have been taken from outdated documents (although for some things, i.e., “plasma” I’m really not sure what the source could have been).

For the other projects, the “consensus mechanism” column is all over the place. Raiden is listed as POS, when it is in fact a channels protocol. Fuel is listed as “POW”; fuel is an optimistic rollup protocol — presumably “POW” is meant to imply “derives consensus from its L1, which itself is POW,” but that’s inconsistent with the rest of the column (and confusing). Things like “non-custodial” (KChannels) and “ZK-SNARK” (matter labs) don’t really answer the question of the column. (There should also probably be some reference as to whether the protocol is an L1 or an L2, as it’s generally hard to make sense of this column otherwise.)

R.e. TPS, comparisons across projects are always going to be problematic given the different underlying assumptions, but even that aside, some of the numbers just don’t make sense at face value. I.e., Connext is said to have a max TPS of 1,000 and KChannels of 10,000; both are channels, and thus (optimistically) aren’t bottlenecked by L1, so it’s unclear why the numbers should be different from each other, and why they should be lower than any non-channels protocols on the list (if anything, shouldn’t both be infinite?). I don’t know anything about Everest, but I’d wager it can handle more than 1 TPS. Etc.

I maintain that the “cost of 100 transactions” should be gotten rid of, as I don’t think there’s any meaningful way to assign it a constant value. See https://github.com/raid-guild/indexed/issues/125

In short, the site still isn’t ready to be live. Many of us put a lot of effort into trying to educate the community about scaling protocols and their various trade-offs, so it’s frustrating to see things like this be put out into the public. I’m all for crowd-sourced initiatives, but that only works if the feedback is incorporated (!), and even that notwithstanding, somebody ultimately still has to do some basic due diligence and take responsibility for what ultimately gets published (even if for no other reason than the fact that people aren’t always the most objective parties in describing their own projects, self included.)

chair28980 commented 3 years ago

@DZGoldman

Thank you for your attention to this and re-submission, first off I have updated the Arbitrum docs on the site to the information you submitted via the link.

We recognize the wild inconsistencies in the docs and we are working to correct the errors. Part of our plan is reaching out directly to members of each team, so thank you for taking the initiative. If you see anything further, for Arbitrum or otherwise, please open another issue and give us the feedback.

I agree with your thoughts on "Cost of 100tx", TPS, and comparing of data between chains, perhaps our approach of a data table on the front page is not optimal. What would you like to see on the homepage of the site?

I apologize for the misrepresentation of the project(s) thus far, and will continue to work to true up all the data. This weekend will see a lot of updates, once we feel more confident with the site we will begin publicizing, until then I will lean on feedback from devs to guide the direction of the site.

If you are in contact with other dev team member who haven't heard from us yet, you can encourage them for similar feedback, our goal is for a curious/interested dev to use indexed.wtf as an initial information gathering source, this should very much be a community driven evolution.

chair28980 commented 3 years ago

@DZGoldman also, I've left the crowdsourced submitted links for now, would you like to see anything added/removed?