Design options and use cases for Oracles

lock9 commented 5 years ago

According to the accepted proposal https://github.com/neo-project/proposals/pull/78

List use cases Discuss what is the best way to implement it (best design) Discuss the scenarios we should cover Discuss security issues we should be concerned Implementation design

Edit: Implementation design proposals without images will be punished with a thumbsdown (👎 )

igormcoelho commented 5 years ago

@lock9 if you prefer to group here all decisions for oracles, I'll help you.

This last issue I opened has nearly a complete proposal for oracles, in two different execution modes (trusted and non-trusted), and hiding CN IP address, which must be a requirement in my opinion. https://github.com/neo-project/neo/issues/932

The proposal for two different operation types on oracles is here (I'll close that one, to work here): https://github.com/neo-project/neo/issues/798

Proposal for TransactionAttributeUsage.Url: https://github.com/neo-project/neo/issues/769 (this one was independently proposed by Erik and Shargon in Native NEP-5 big PR). This is already implemented.

I couldn't find the original proposal of oracles, which was by Shargon and Erik, some time ago.

@doubiliu is very active there and has given priceless contributions for the design as well, so let's summon here as well ;)

igormcoelho commented 5 years ago

I'll quote my last comment there: (from #932)

@doubiliu interesting analysis. In fact, like you said, this would indeed affect synchronous process, but we can (and must) do it in a completely asynchronous way (just like transactions). I'll present two options.

Option A: dettached transaction + tx validated oracle data Before some transactions are selected for block, the CN must verify that its oracle data is already processed (remember that the outer ring will send its result over p2p, for every tx they have on mempool). So, when a tx is submitted from a CN, it must already contain the oracle data processed (by the independent outer ring), otherwise it won't even be a candidate for block (will keep waiting on mempool). What may happen is that a CN proposes a tx, with validated oracle data (according to the outer ring, passed by p2p), but other CN only have tx (not outer ring p2p oracle data). In this case, these CN would have to do a similar request of RequestTransaction, but like RequestOracleDataForTransaction, and this information would need to be received during block time. At this moment, it's not time to execute the oracle itself, just to receive the same previously validated oracle results that the speaker node already has (flowing through p2p). This only affects oracles that require trusted execution (if not trusted mode, then problem is solved already, as long block proposal includes oracle data). If this oracle data is not received in time (on trusted mode), it's the same effect that the tx hasn't been received in time, thus invalidating the block. We can design a payload that flows together with transactions on p2p, so this increases the odds that, when someone receives the tx, it already receives the validated oracle data (but risks of invalidating blocks on trusted mode due to non-existing oracle data will still be possible).
Option B: Attached transaction + tx validated oracle data (tx with oracles must be routed by oracle ring) Another option is: oracle data could be a pre-requisite as well, meaning that tx with oracle can only flow through p2p when oracle data has already been validated (this would work for both trusted and non-trusted mode). This means that tx with oracle data would be submitted first to the oracle ring, get the oracle data, and then it is automatically routed through regular p2p (this means that every CN that receives this tx, will also already receive the processed oracle data). And since the oracle ring itself will route the transactions, this reduces the risk of someone trying to cancel specific transactions, because they don't like oracle result (like lottery contracts, for example). This attack will still be possible if oracle ring is completely cut-off from p2p, but as long as this ring has direct connection (the one I mentioned to shargon), the attacker will only know the result when it's already being routed through p2p, and even re-routed if lost, so an attack would be quite hard in this sense.

eryeer commented 5 years ago

As far as I know, we are docking some oracle third-party organizations (such as Chain.Link), and then these third-party organizations perhaps will provide oracle data services to Neo in the future. So is there any relation or difference between the oracle design we discussed here and the docking?

igormcoelho commented 5 years ago

@eryeer I'm not aware of the docking you mention, can you clarify? WIll it require an extra attribute transaction field url, like the one we are proposing? Or can it work without it? How can a dapp user be certain that provided onchain oracle service is trusted?

I'll present my "mental challenge" on the Oracle feature. Suppose we want to build a lottery system on Neo, that depends on random.org website, and will give millions of dollars in prize. Can we build it using the third-party provided? This is what I'm trying to (I wanted to) achieve here.

steven1227 commented 5 years ago

@eryeer I'm not aware of the docking you mention, can you clarify? WIll it require an extra attribute transaction field url, like the one we are proposing? Or can it work without it? How can a dapp user be certain that provided onchain oracle service is trusted?

I'll present my "mental challenge" on the Oracle feature. Suppose we want to build a lottery system on Neo, that depends on random.org website, and will give millions of dollars in prize. Can we build it using the third-party provided? This is what I'm trying to (I wanted to) achieve here.

Of-course we can use the third-party oracle service. There are no conflicts whether NEO has its own oracle service or there are third-parties.

longfeiWan9 commented 5 years ago

@igormcoelho FYI, we are working with ChainLink to build Oracle service on NEO so that projects can use, and Chainlink are working with a lot of big data providers . So do we still need to put so much effort to provide Oracle in NEO by ourselves.

steven1227 commented 5 years ago

@eryeer @lock9 @doubiliu @igormcoelho @erikzhang Before the design, I believe the issue we have to put in the discussion is why we need to build our own Oracle and what is the pros and cons.

eryeer commented 5 years ago

@igormcoelho We can discuss the relevant issues in face-to-face meetings in September.

EdgeDLT commented 5 years ago

@wanglongfei88 @steven1227 I don't think we should be dependent on third party solutions and tokens to offer this functionality.

Chainlink support should expand the options for NEO developers, not replace them. NEO should still offer its own Oracle solution.

WyattMufson commented 5 years ago

I agree with @EdgeDLT on this one. NEO should offer a full solution for developers without relying on the existence of a third party. Having Chainlink support to supplement the built in NEO oracle service would be ideal for developers.

igormcoelho commented 5 years ago

My point is just if ChainLink can provide such a guarantee that I mentioned. If it can, we can adopt it, I'm not against third party solutions.

Let me clarify the challenges I see: 1) CN cannot be exposed. Thats why I'm discussing outer ring oracle nodes to access data relay tx. I dont think CN should even have Internet access, so how could them directly communicate with thirdparty? Couldnt a failure in thirdparty cause all CN to be hacked? There's block proposal time issue too.. is it enough time for all CN to request thirdparty to process every tx, in few milliseconds? 2) If its not direct contact of CN with third party, then thirdparty should directly relay all oracle-enabled tx. For example, if someone makes a poker-dapp game move, and doesnt like the random outcome,could it not send the tx and try again? I dont think so... so, all oracle enabled tx should guaranteed to be relayed,or fail by block time expiration.

If that works, or solution is even better, that's good for us,I'm just raising dome concerns on oracle design that werent raised before,and I think they're important.

lock9 commented 5 years ago

We need incentives and the features needed for these providers to work with neo. The ability of them being paid using smart contracts.

Like people already mentioned before, we need some kind of delegation, because CN won't access the internet, it will probably count on some sort of proxy to make these requests. We need some sort of 'oracle fees'. Maybe, for these fees, we can even allow them to be paid using nep-5 contracts, like @vncoelho mentioned in #978.

We need to allow Neo to support different providers and make this "profitable" for them (allow Neo users to pay them fees).

igormcoelho commented 5 years ago

We can discuss the relevant issues in face-to-face meetings in September.

Sure @eryeer , we want the best of the best for Neo ;)

EdgeDLT commented 5 years ago

Consider the example set by NeoFS. There is an outer ring of storage nodes paid in GAS (creating incentives for NeoFS network participants and NEO holders), and an inner ring of nodes that handle auditing/accounting.

The same model should work for an oracle network, except network participants would be offering "trustworthy" request data for review instead of providing storage space. The inner ring would be responsible for handling payments/punishment, validating request data, and somehow making the validated information available to CNs.

igormcoelho commented 5 years ago

In fact, I didnt know NeoFS work like that :joy: interesting.. when I say "outer ring", for me is just an "external ring" of nodes, so we wouldnt need internal ones (btw, NeoFS is even more complex than this proposal, congratulations to NSPCC).

EdgeDLT commented 5 years ago

Yep, in this case I guess CNs or contracts act as the 'inner ring' seen in NeoFS.

I just think it is important that NEO provides its own infrastructure to handle transaction routing or to let people participate by running nodes (ideally with incentives powered by GAS). I don't want to see a situation where important features like oracles for contracts are unavailable to NEO developers that do not want to trust a third party project and its network or token.

The way I see third parties like Chainlink fitting into the picture is by connecting as a part of the NEO oracle "external ring." It's up to them to build a product that will convince developers to use those nodes over the native NEO infrastructure. This is what I mean by avoiding dependency; NEO should provide an implementation that these projects can then extend on their own terms.

igormcoelho commented 5 years ago

The only thing I'm afraid is thirdparty not providing Oracle service with one block finality (by third party externally reading and writing on the chain).. block finality is not common, other chains dont have it, so it might be ok for them, but Neo has it and we need to take advantage of it. We need to better discuss it, I dont know the details.

pappas999 commented 5 years ago

The concept of a 'baked in' Oracle into the protocol is a great (and ground breaking) idea, and in my opinion if it can be pulled off properly, is an important feature to have for a number of reasons (which Edge touched on). Here's some of my thoughts:

Most platforms aren't suited to have this in the protocol due to their architecture. NEO is suitable for this due to the lower number of consensus nodes coupled with the dBFT consensus mechanism. However as stated above, there are some careful considerations that need to be taken into account re: internet access, block times, security and effect on consensus etc. If it's done, it should be relatively easy for a developer to make use of this Oracle for simple requests, and be able to pay for the request with GAS.

Chainlink has built a great decentralized Oracle network which is quickly becoming the standard for getting off-chain data to all blockchain Smart Contracts. However not all use cases require this level security, decentralization and complexity required to implement.

ie lets say a Smart Contract on NEO pays millions of $ out insurance to farmers based off weather events, then you'll likely want to use multiple sources of data for weather rather than just 1, and have the results aggregated off chain and passed to the Smart Contract. This is where something like Chainlink shines, it gives you confidence in the data due to its ability to pull data from multiple sources & oracles, have it aggregated and with nodes staking tokens and having penalties for if they are found out to be acting bad.

However if you are a simple small startup or developer building a game dApp that needs to connect to a custom web service you've built and deployed on AWS, then it would be better if you could simply call the service from within your Smart Contract without needing to go through an external network, obtain & consume LINK tokens etc.

If NEO can offer both options to developers, I believe it will greatly help adoption from developers, individuals, startups all the way to big enterprises, because it covers all bases from the quick, simple & easy, through to high value scenarios where a very high level of security and decentralization is required

longfeiWan9 commented 5 years ago

I like the idea that @pappas999 proposed.

Let Chainlink team to handle the complex Oracle service which takes decentralization, security and efficiency into account, Because that will require lots of effort, expertise and partnerships.

Then NEO could just provide Simple Internet Access to developers or projects without worrying decentralization , Byzantine nodes or malicious users, etc. It is just a simple web access service or resource developers want to use and access. I thought this is also the original propose.

igormcoelho commented 5 years ago

Then NEO could just provide Simple Internet Access to developers or projects

That's what this proposal is about @wanglongfei88 :joy: it's challenging to offer such a simple service. My proposal tries to made this as simple as possible, to be honest.

corck commented 5 years ago

A use case that I (us from Novem) would implement right away when we have oracles.

Autoconversion of tokens. At Novem we give 2% discount on purchase of our NNN token if its paid with the NVM token. Both prices fluctuate, NNN tokens relate to the Gold price, NVM is market price on exchanges. So I would want our NNN purchasers be able to send NVM tokens to my conversion smart contract. The oracle then gets the current price for the NVM (e.g Coinmarketcap API) then the NNN price (from a NOVEM API). It then calculates the exchange rate with the additional discount and converts it right away.

There are a few more use cases like a savings plan, auto portfolio allocations (like having a BTC / Gold portfolio, that auto adjusts to keep a certain exposure to both assets), that comes to my mind, if we can get external pricing.

pappas999 commented 5 years ago

A native Oracle could also potentially be used to obtain the $ Price of NEO or GAS, so instead of say changing 100 GAS to deploy a contract , you could say it costs $100 USD always, and then use the Oracle whenever a deployment is done to enforce the correct amount of GAS, instead of having a hard coded static fee

corck commented 5 years ago

Agree with @pappas999, something I'd love to have as well.

We really see an advantage of having free transactions with NEO, but as this will be changed we are thinking of covering the transactions costs through our transactions fees (we have in our NNN token), to have the users of our Gold token to not deal with purchasing another cryptocurrency (GAS), which will make it much more usable than other Gold tokens. For this to achieve we need to get the price of both tokens and the transaction costs, also if there's a spike in Gas price, it might not be sustainable to cover these costs, then we need to stop it.

Its still an idea and might not work out as well, but ideally there would be a way to cover the gas costs using our own token.

igormcoelho commented 5 years ago

@corck interesting point on NEP5 as network fee. please see this discussion on the topic: https://github.com/neo-project/neo/issues/978

Tommo-L commented 5 years ago

If the out ring oracle nodes can be directly interconnected, the consensus transmission process can also be accelerated. 😂

igormcoelho commented 5 years ago

I'm afraid we cannot do that direct connection, for security reasons. Someone hacks an oracle node (which has public ip), you will expose CN ip, and worse, get a direct connection to it :joy: It would be nice if we could, this I agree... like directly connecting to CN :rocket:

shargon commented 5 years ago

We are designing different solutions, in the leading one we include a new actor, masternodes/oracles, these nodes will perform the oracles functions.

And we will sign the result of the request involved in a new transaction, OracleTransaction, with the signatures of this nodes, and the downloaded data

The CN won't accept OracleTransactions without M oracle signatures

igormcoelho commented 5 years ago

@tommo-l funny thing, my vision of an external oracle is slightly different, I'll try to draw here (not as beautiful as yours :joy:) its like a heavily connected subcycle, or perhaps a graph clique (although not necessary). Oracle links could be prone to attacks and DoS, but it doesnt matter too much, since it wont compromise regular non-oracle operations.

Green nodes are CN.. Red are Oracle nodes... the blue are regular p2p nodes. Black connections are permanent, yellow are regular p2p connections to.unknown neigjbors.

Screenshot_20190821-153050_Rascunho

lock9 commented 5 years ago

@shargon I liked the OracleTransaction. Do you think we need to store the response every time? I think that we should start with some basic internet access. What if these OracleTransactions can be started in one block and finished in another one? Maybe we can have a callback in the smart contract, used when the oracle transaction 'arrives'.

shargon commented 5 years ago

This transaction will be on the oracle pool and only will be stored in the block when oracles will confirm it, we can create a fee field for this oracles nodes (ON)

shargon commented 5 years ago

oracle transaction is like a envelope of the original TX, and the signature of the ON, the memory pool should wait until this signatures are enought for move from the oraclePool to the memoryPool, and then will be safe the execution of this TX

EdgeDLT commented 5 years ago

@shargon I like that idea, with Oracles providing signatures before the OracleTransaction will be accepted by CNs. As I understand it, it would mean that the P2P nodes will help propagate these transactions, which should avoid exposing CNs right?

I have a few questions about that though:

Would one node propose a value (say a contract requests the price of GAS, and one Oracle claims a price of $1.65), and the others sign if they agree on that value? Or are OracleTransactions only created from aggregated and averaged data that the Oracle Network has achieved consensus on?
Would they all need to retrieve that value from the same source(s), or can any source be used and all that matters is the consistency of the returned values?
What if one node sees $1.66, or $1.69? Would that be considered within an accepted range and be signed? How do we determine such limits?

igormcoelho commented 5 years ago

@shargon I dont think.we need oracleTx. Oracle machines will have IP exposed, right? So they can have direct link between themselves, handle oracle computation, collect signatures, and only then relay this signed tx to the p2p world. Whats the advantage of using regular p2p to distribute partially signed oracle tx ?