sneg55
commented
4 years ago goal is to keep idiots from investing in DeFi
I think idiots have more chances to make a bad investment into "DeFi coins" popping up, eg. https://coinmarketcap.com/currencies/defi/ rather than chase for 10% APR in DeFi lending.
Smart contract risk goes down when AMOUNT OF T I M E AND M O N E Y goes up
Disagree, with upgradable smart contracts any update may bring critical bug.
"Centralization Risk" is so crypto nerdy it hurts
💯👍 users outside of crypto twitter don't care.
MatthiasEgli
hughkarp
marsrobertson
tayvano
Preface: I love what you guys are working on. Def much needed. But, realistically, if this very necessary thing is going to keep a diverse range of user types informed of the risks (or even just maybe prevent a few people from throwing fat piles of crypto at random defi platforms that have been overhyped on twitter and underaudited in reality), I think we are going to need more. I'm going to dump it on you. Please don't take it personal. This says more about the space and I am all too pleased to see you take this hard issue head on and move so fast. Your product is needed. Also, please forgive how hot-messy this is. 🚒
What is the goal? Who is the target audience?
Right now DeFi is a fucking dumpster fire 🔥 and any market movement will be like throwing gasoline, old christmas trees, and a pile of overly-hairsprayed hair on top of it 🔥🔥🔥🔥🔥. Also, it's hard to build a product for everyone. Yay!
Questions: What's the end goal or use-case that DeFi Score imagines having the greatest impact? Is a one-size-fits-all number really giving the user what they need to make a somewhat—anywhat—informed decision? Can you use same input for a tailored output for different user-types?
If your goal is to keep idiots from investing in DeFi then you may just want to perfect a very beautiful icon that is universally recognized. Like the poison icon. The radioactive icon. Or the Beware of falling rocks! road sign. Or all of the above. 😅
If the goal is to make users relatively aware that investments into DeFi platforms may... carry risk + carry more risk than just holding crypto + carry more risk than the same action/position in the traditional world (lending, leveraging, shorting) + inform them that not all defi products are created equal.......it may be worth considering how you capture not just the risk, but the relativity of said risk to things a user may be more familiar with.
If the goal is to give product creators access to a score that they can display in end-products then you can de-prioritize a beautiful site for end-users with the scores, dashboard, etc. and instead focus on selling to products who touch these end users. Expend more energy on giving these products access to a diverse range of information + good documentation + good examples + good case studies + empower these products to make their own specific choices as it relates to their product and their user demographics. For example, a mobile wallet targeting noobs may display a simple red-orange-green color system. Multis (a multisig interface for treasury management) may show as much information as possible The advantage of taking this route is that the end-product should (hopefully) know their users best. This then frees you up to create a product that serves a very wide range of user types without actually having to literally serve all those users directly. (Because, take it from me, it is NOT fun to build a product for all the user types!)
Smart Contract Risk
Misc notes that I couldn't fit elsewhere:
the number of audits I've seen where the project didn't implement all, or even most, of the recommended fixes is fucking terrifying. Just because something has been audited, doesn't mean anyone read it. This increases risk as not only is there an issue, but that issue is known and literally spelled out in the security audit for anyone to find!
Similarly, a very dangerous time for smart contracts/protocols is after an audit has been complete, but before fixes have been implemented / funds have been moved to a fixed contract.
Similarly still, even speculation about the security of a smart contract increases the risk of a bug being found in that contract. For example, the DAO bug was almost found by Gun shortly before the attack. He detailed the issue however hadn't connected a little piece and so the issue wasn't actually an issue. Days later, the attack starts, and lo and behold, if Gun or others had spent a bit more time or their brain had been working a bit differently, they may have found it first.
Smart contract risk decreases as AMOUNT OF TIME AND MONEY increases
I think this is really, really a huge hole in your current analysis. As someone pointed out, since most contracts can be updatable in a myriad of ways, time does NOT specifically reduce that risk. However, it does reduce the risk of attacking the contract directly, economic stuff going haywire in the contract, etc. You already separate security of smart contracts from admin/updatability of contracts so I'll just be explicit that I'm focused on the contract attack vectors themselves, not the admin ones.
Background:
The absence of security audits and formal verification increase our certainty about the risk of a product far more than the presence of them. For clarity: If flipping a coin is 0 and not having an audit is -100, then having an audit would be a 10. 10 is way better than -100, but not much better than 0.
Therefore, any smart contract without a security audit is far more likely to be a scam or have creators who have such low appreciation for security that it is almost certainly insecure. Regardless of all else, this should be weighed very heavily.
A smart contract that has an audit...could totally still be hacked / broken / manipulated. Therefore, this should be weighed less heavily.
(Grain of salt disclosure: I'm obviously still suffering PTSDAO and PTSParity#1 and PTSParity#2.)
Things that do increase certainty around / decrease probability of Bad Things™ happening
How battle-tested and hacker-tested is a contract or system? This is the reason we trust the Gnosis Multisig more than the Gnosis Safe even though they were both created by one of the most diligent teams in the space. In the same vein, if you deployed a multisig and put $1m in it on 1/1/2017 and it wasn't hacked by 1/1/2018, I would be more confident in that contract than the same one being deployed on 1/1/2019 and surviving until 1/1/2020. This is because the amount of hacker eyes and the sophistication of said hackers was greater in 2017 than in 2019.
I am more certain that a contract won't be hacked/broken when...
There are more funds held by said contract. This is relative to other crypto contracts and also relative to other things a hacker could do to make a quick buck. Also, having big honeypot in a single contract is more likely to attract more nefarious eyes than a pass-thru contract.)
It's been in production use for longer. Duh.
It's survived periods of high volatility in short amount of time; it's survived large drops or long-term bear market conditions; it's survived large gains or long-term bull market conditions. (note: only applicable in some cases, e.g. DAI)
It is more popular on a social / PR level. Does everyone in the ecosystem talk about it (Compound)? Do people outside the ecosystem know about it (the DAO)? Or does no one even know about it (all the other random contracts out there)?
It is more used. Have there been a lot of transactions through the contract? Are those transactions by a relatively diverse set of people or market conditions? How big are the transactions? (This one specifically helps more with contracts that fail due to game theory / bad economics / etc.)
It hasn't changed, nor have admin contracts changed, nor have addresses changed, etc. Example: DAI: Team, brand, naming, everything looks the same to a layperson. But hey, you know, it's actually been in production for like....70 days. 🤷 The Parity Multisig #1 & #2: had held hundreds of millions for a long time, had been audited, was a slightly different version of the original foundation multisig which still holds the EF's ether (safely) today. But then they modularized it to save gas. And that was enough to make a vulnerability exploitable.
It's been hacked before. There is a ton of literature on this sort of subject if you want to dive into this so I'll try to be brief. Essentially there are two viewpoints and I don't know which I agree with more:
If something has been hacked + fixed, it's more likely to be secure. Example: Monero vs Zcash. Rationale 1: you learn from experience and mistakes won't be repeated. Rationale 2: code always has bugs and if you haven't found any it's only because you haven't found them not bc they don't exist.
If something has been hacked, it's less likely to be secure. Also, other contracts by same teams are less likely to be secure. Rationale: the company/people don't know how to be secure, there is bad company culture, immaturity, bad QA, contract is too complex, etc. Examples: Parity (obviously). The flipside is Gnosis. Though there still is time before we know if the Gnosis Safe will hold up to the security of the original Gnosis Multisig, their track record has been good so far.
Somehow, these must be captured. I believe this is the #1 factor that will move smart contract security risk around. I would even say that an unaudited, unverified contract by a non-name team that has held billions for a long period of time is more secure than an audited, formally verified, blah blah blah contract by a known team that's held $500k for all of 2019. (Assuming both are non-upgradable, of the same nature, etc.)
PS: I am not alone. Ameen phrases is thusly regarding compound:
https://medium.com/@ameensol/what-you-should-know-before-putting-half-a-million-dai-in-compound-fafdb2645f77
Everything is relative!!
Do I know what it means if
UnknownDeFi#1has a higher number thanUnknownDeFi#2? Probably not, because I don't know what either really are or what the numbers really mean. However, I probably have some sense of the risk of holding crypto vs risk of holding USD vs investing in stocks vs investing in gov't bonds.If you label a
gov't bondas a 1,stocksas a 2,UnknownDeFi#2as a 90, andUnknownDeFi#1as 95, andGivingAStrangerAllMyCashToHoldas a 100, that's far different than justUnknownDeFi#2as a 90, andUnknownDeFi#1as 95.Consider using or providing icons, pictographs or words
Numbers are meaningless without a lot of context. However, things like these capture relativity and and digestible at a glance:
or...
"Centralization Risk" is so crypto nerdy it hurts
Yes, how contracts are controlled, managed, updated, fed data is potentially highly risky. These are necessary categories!! BUT! Don't classify them as "centralization."
When a crypto-native looks at this type of risk and assigns a category of "centralization," it makes sense. But, when you start with "centralization risk," this is NOT what comes to mind. When I think of centralization I think of The DAO vs Compound vs Blockfi. I think about whether I trust smart contracts more than a custodian. Etc.
May be worth renaming protocol administration and oracles to something else. 🤷
And, since we are on the subject...
Admin/access/upgradability risk is very diverse
The Taylor Hack: "Somehow the hacker got access to one of our devices and took control of one of our 1Password files."
The Bancor Hack: Hackers accessed a multisig wallet used to upgrade smart contracts and withdrew the money mostly in Ether.
Platforms that have a big red button to halt, update, etc. While this introduces the risk of a nefarious third-party gaining access to an admin key and changing shit, it also reduces the risk of total catastrophic loss if a hack does occur. So that's a thing.
Platforms that don't have a big red button. There is no way to stop, pause, recover, save, anything in any condition, including bad code, hackers, something breaking, bad economics, etc. So THAT'S a thing.
What is the risk of the multisignature contract that protects the upgradability of the platform? Kidding....but not really at all. Imagine if Bancor's admin contract was a Parity multisig and now Bancor's ability to update their shit is locked. Oopsies! Compounding risk is fun!
Compounding Risk
How would one start to be able to calculate the risk of things combined?
Multisig example detailed above
Risk of holding DAI vs risk of lending DAI via Compound.
If I take my hard-earned ETH, exchange for USDC on Uniswap, put the USDC into Compound for cUSDC, then mint the ETHRSIAPY Set on TokenSets...
dontsayitdontsayitdontsayitdontsayit ARE YOU SERIOUS WHAT ARE WE GOING TO DO ABOUT SYNTHETIX?! OMG, or pooling sETH via Uniswap. Ahhhhhhhhhhhhhh.....🏃no, run AWAY my emoji dude RUN AWAY!
And while some tokens are purely speculative and their risk is very akin to traditional market risk, some (DAI, sETH) do heavily rely on the security of the overall platform. A good test would be to ensure that the risk of standard-speculative-erc20-token is not the same as sETH and neither are the same as ETH. Luckily the market risk (e.g. price to 0 bc entire team and community died suddenly) cancel each other out as they exist for all the tokens so you can focus on the other risks except oh yeah fucking stablecoins. 😉
People / Team / Culture
I don't know how this fits in exactly, but it does. If you talk to two teams in the space you will see differences in priorities, specifically UX vs security. Good example is Gnosis vs Argent. Gnosis is willing to go to market slower, be more diligent, perfect. They are scared. They have crazy internal processes in place for security things. Argent is...just not that. They prioritize getting users, having best UX possible. Which you prefer is subjective, but as the provider of an objective-ish DeFi Score, the emphasis will have to be put strongly on security over UX.
I have a lot of ideas around this and I'm sure others have more, but I'm not sure there's a way to capture this via an algorithm as there is a lot of subjectivity. Ideas:
Look at how the team responds to security audits or a person reporting a security issue. The best example of this is Coinomi. 2017: i ignore you on github, i ignore you on reddit, i yell at you and call you fud weeks later, i distract by saying "well at least were better than another insecure wallet!" 2019: ignore, deflect, hid old tweets, denies, blames victim, yells, distracts.) (source: https://bitsonline.com/coinomi-vulnerability-respond/, https://www.reddit.com/r/CryptoCurrency/comments/av7gfi/warning_coinomi_wallet_critical_vulnerability/). I don't know if Coinomi is secure or insecure, but I sure as hell know they they don't know either and will yell any anyone who tells them they have a problem. I can also tell you white hats and gray hats start looking mighty dark once you treat reporters like shit.
How obsessed and paranoid about security are they? Talk to Robert (Compound) for 15 minutes and he will readily advertise that there will always be a non-zero risk for code. This is good. People pointing at their audits if people ask about risk....not so much...that's just deflection. Check out some recent podcasts with Ryan Selkis on what they are doing at Messari. There may be a way to capture universal info by classifying the responses to certain questions. It may just throw a red flag. I don't know.
Culture - how they update, when they update, what their github looks like, what any bounties / audits have found, how quickly they respond to a vuln report, how they respond, etc. Is there a correlation between a bunch of critical bugs in initial audit and bugs in the future? Is there a correlation between a bunch of little bugs in the audit and a critical bug being found in the future? https://github.com/solidified-platform/audits may be a good place to start with this data.
Bonus Points: Normal Usage/Market Risk
When I consider my users, one of the biggest risks in integrating defi platforms is whether or not the user actually understands the very-well-studied market risks that occur in any market. In the traditional financial sector, the people investing do know that an asset could go down and could go up. In crypto even this most basic fact isn't necessarily known. More worrisome, some think they know but they really don't.
I categorize these risk separately than the more extreme risks of a contract going to zero or nearly-zero. The events that cause these risks to happen, happen regardless—it's just a matter of luck whether you lose or win on any given day. This includes things like...
The market going up or down, your position changing for the worse, that you may have been better off just holding ETH, or that you lose money because of your position. These will not destroy the contract or system, but they may destroy an individual.
Risk of having your individual position wiped out. Again, this will destroy you but not the system. Things like liquidation come to mind.
Example: compare a normal lottery to Pool Together. Playing the lottery has a very high risk of losing your entire "investment". However, with pool together, that risk is ~0 (assuming everything else works as intended.) If your risk score puts the risk of a traditional lottery (but onchain/with crypto) to a flawless implementation of Pool Together, the scores should be different.
There's like tens of thousands of people who do nothing but research and analyze these types risks in order to make the right calls in the real world so it's unlikely you're going to figure this out on a huge level.
BUT perhaps a score could be given to just show whether there's a strong likelihood of retaining initial investment, gaining, losing, or 'your guess is as good as mine, crypto is volatile, shrug.' This would at least help differentiate between gambling vs pool together vs lending vs 100x longing. Right now, I worry that theoretically, a gambling defi thing could have the same risk score as lending which doesn't feel right.