neo-project / proposals

NEO Enhancement Proposals
Creative Commons Attribution 4.0 International
136 stars 113 forks source link

Transfer to Transaction NEP-TTT #137

Open igormcoelho opened 3 years ago

igormcoelho commented 3 years ago

Summary: this NEP called Transfer to Transaction tries to provide a practical manner to redistribute assets during Verification time on transaction, thus allowing implementation of practical mechanisms for "free transactions".

[EDIT 1] - This doesn't affect NEP-17 in any case. At the time of writing, this is mainly intended for adoption only on GAS contract, as NEP-17 and NEP-TTT. [EDIT 2] - Two methods are proposed in this NEP: scheduleTransfer and finalizeTransfer.

[ORIGINAL PART]

=====

Transfer to Transaction

This NEP considers a transaction as a valid (and temporary) token holder (as HASH256 identifier, not usual HASH160). It can be done with operation "contract.scheduleTransfer(from, target_tx, value)" that launches a "promiseNotification" (consumed before actual contract invocation). Contracts can easily extract funds from transaction, for example, in a mint operation. Neo system can also consume funds from Transaction, for example, GAS funds to cover fee for operations. User can also specify a "contract scope" or "group scope" as the temporary holder (instead of global "transaction scope") and funds are consumed according to the scope.

Example 1 (usual operation):

Example 2:

This "request" operation at ContractY can do some interesting thing such "redistributing" GAS for user operation, in that specific scope, with some simple rule as:

This means that giveaway operation only works after few blocks, and with limited gas supply (limited per block hold time and user shares on that contract).

igormcoelho commented 3 years ago

Related to https://github.com/neo-project/neo/issues/1468

erikzhang commented 3 years ago

So these two transactions must be included in the same block?

igormcoelho commented 3 years ago

The idea is that this is a single and atomic transaction.

One way to implement this is to put some new verification script field on tx header (I'm avoiding this approach) Another way is to have the script part that emits the GAS.scheduleTransfer(sender, GetTransactionHash(), value) to be embedded in a co-signer (I prefer this one, because it's supported by current mechanisms) So user attaches a sender witness, and another "transfer witness" that generates collateral effect as a "promiseNotification" (just like a notification, but that vanishes as long as invocation part of transaction begins).

With this, Neo system just needs to catch the "promiseNotification" from GAS for "transaction balance" (and maybe the specific contract scopes of that witness), and this can be done deterministically (I hope so), before the transactions is put on block (the same as fee calculation currently being done). This way, we don't interfere much with Neo system, and for user, it just needs to implement NEP-TTT together with NEP-17:

function scheduleTransfer(HASH160, HASH256, BigInteger)

And maybe another way (not sure now if it's necessary): function consumeNotification(NotificationObject) - called after notification is "consumed" (begin of invocation part)

erikzhang commented 3 years ago

So this is a solution to let the contract pay for the fee?

igormcoelho commented 3 years ago

So this is a solution to let the contract pay for the fee?

I think so @erikzhang

erikzhang commented 3 years ago

Then why not use the contract hash as the sender of the transaction?

igormcoelho commented 3 years ago

:thinking:

igormcoelho commented 3 years ago

Then why not use the contract hash as the sender of the transaction?

[EDIT]: sender means signers[0]

That's a very good question. Using the sender looks nicer indeed.

I think that using the balance from sender requires permission, and it could indeed provide permission to spend its GAS assets. I just don't know how it could give "partial permission" over a limited amount of assets (imagining some "unbounded" verify() returns true). Another trick is to prevent the actual movement of assets, to forbid non-determinism (that's why I think we should do all this on Witness processing level with special function "scheduleTransfer", not real "transfer").

But I think you're right, Neo System could automatically invoke this "scheduleTransfer" directly from sender (as a contract), to launch "promiseNotification" and guarantee the existence of GAS to make whole its future operation, and store it on current transaction (this is much easier for everyone as no new script is required, all automatic). One trick of using "sender" is that maybe the remote contract wants to give partial allowance according to the "user" (but who is the user now?). That's why I still miss a role of the "sender" (who actually wants that tx processed) and maybe a "payer" (which is typically the sender, but not in this case).

igormcoelho commented 3 years ago

Let me just highlight the most fundamental points trying to be accomplished here, in my opinion:

roman-khimov commented 3 years ago

This NEP considers a transaction as a valid (and temporary) token holder (as HASH256 identifier, not usual HASH160).

IIUC this requires changes to NEP-17 and if we're using "transactioned" GAS for fees then it also requires senders to be 256-bit wide which is quite invasive.

"partial permission" over a limited amount of assets

Verification context has access to transaction and it can get fee values to decide whether it wants to allow spending a particular amount of GAS for fees.

That's why I still miss a role of the "sender" (who actually wants that tx processed) and maybe a "payer" (which is typically the sender, but not in this case).

Isn't it all controlled by witness scopes?

funds are stored on transaction, for direct usage, with limited scope (respecting the given witness scope)

I think I don't quite understand this "stored on transaction" part. If actual contract storage isn't affected then it's a kind of "ephemeral" storage and this then raises questions like what happens if 0.5 GAS was requested and 0.4 spent, what transfer events should be generated for this storage/spending (especially for fees). Also IIRC notifications are forbidden at the moment in verification context.

this makes it easier to consume one-time from transaction (such as in minting)

But do we have any problem with that? I think we have any minting/staking/depositing case covered by simple NEP17 transfers with additional data to contract's address, then the contract in question can do anything it needs to in onNEP17Payment handler.

this makes it possible for contracts to give away GAS that can only be used within a given scope, meaning that it won't be used to sponsor third-party operations

I think this case can be perfectly handled by Notary subsystem from neo-project/neo#1573. Contract's address can still be used as a sender (but some other address can be used if needed), but user will create an incomplete transaction and then the decision on whether this transaction should be completed or not can be made by dApp backend instead of (inevitably limited) contract's verification method. Backend can have some state-dependent logic for which addresses can use this feature and how much GAS they could spend, it can also parse the entry script and for example only allow some specific calls to be made.

igormcoelho commented 3 years ago

Thanks for taking the time to evaluate this proposal @roman-khimov. I agree that neo-project/neo#1573 is an interesting approach, specially as it deals with P2P operations, so I don't see any conflict with this proposal here. This aims at providing temporary storage at current transaction and allow scoped-acess to this temporary storage. I'll try to clarify some points here and presenting a possible "implementation-driven" approach.

this requires changes to NEP-17 ...

This is important: NEP-17 isn't affected in any way (and certainly no address change!). The main intention at the moment is to having GAS contract implementing NEP-17 and NEP-TTT (an extra method), although other contracts may benefit from it as well.

I think I don't quite understand this "stored on transaction" part.

That's the core of this proposal, I'll try to clarify. Currently, user informs that tx "fee" on GAS will be, and then Neo System automatically takes this from user "GAS account" and uses this to makewhole its operational costs. The idea here is to create a memory map/dictionary like this (this represents the "promiseNotification" I mentioned eariler): (TransactionHash, ContractHash, Scope) -> Data

Since data can only be read from inside Transaction itself, TransactionHash part can be ommitted on practice (not a global information, not even block-level, just single-transaction-level).

[EDIT]: sender means signers[0]

The workflow I imagine is:

Note that if sender had funds, these would be allocated according to its own scope, for example Transaction[GAS.id, GLOBAL] if they can be used anywhere during invocation (same logic to its witness).

then raises questions like what happens if 0.5 GAS was requested and 0.4 spent, ...

Another interesting feature is that we can have an extra method (like I mentioned before, finalizeTransfer), to be called after Transaction processing. This method would ensure that Transaction ephemeral storage is cleared after execution... meaning that it could be used to claim back assets put on Transaction. Something like GAS.scheduleTransferFrom(GetTransactionHash(), ContractY.id) (with no value, so it takes all available content and clears transaction memory).

But do we have any problem with that?

No, we can solve minting using OnPayment strategy. But it's powerful to have some "shared space" for assets during contract invocation, so that one can give assets to another one, or even cover system fees directly (it could even re-fill system fees during runtime, but we decide how much flexibility and complication we want to provide).

Does it makes it clearer or more confusing? :joy:

igormcoelho commented 3 years ago

@erikzhang when you have some time, please take a look at the workflow above. Afterall, I don't think using sender solves the problem. The issue I see with using sender as a third-party GAS provider, is that it's not clear to who it is providing GAS to (but I agree that's a problem that needs to be somehow solved by the verify() logic, but personally I don't know how to do it). I think that some explicit operation GAS.scheduleTransfer(ContractY.id, GetTransactionHash(), remainingFee) could explicitly load Transaction with GAS, and we may also give this ability during verify() method, to be used by any contract co-signer. Another solved issue is the ability to manually give back unused tokens, something that secondary method finalizeTransfer can do (during invocation time).

igormcoelho commented 3 years ago

@roman-khimov please take a look at the workflow I've put in details. @shargon do you think this works?

General Outline: What do you think of the idea of Transactions holding temporary assets (including scoped-execution GAS)?

erikzhang commented 3 years ago

Asset.finalizeTransfer is invoked after contract execution, to clear non-empty TTT fields (and possibly to give-back unused tokens)

If the account balance is insufficient, it will cause the transaction that did not pay the network fee to be included in the block. This can lead to attacks.

igormcoelho commented 3 years ago

If the account balance is insufficient, it will cause the transaction that did not pay the network fee to be included in the block. This can lead to attacks.

I've accounted for that already @erikzhang , and this won't happen with GAS as the existing accounting model will be immediately mirrored into this new one. As soon as a GAS.scheduleTransfer is emitted on verification, Neo System is wise enough to deduce it from balance, precisely how it happens nowadays.

This variable gas_amount and this var GasLeft are the ones being "internalized" into Transaction "ephemeral storage".

gas_amount on ApplicationEngine

GasLeft on ApplicationEngine

No attacks can happen, because no non-determinism on balance is generated from this (as no Storage access is performed, besides intrinsic GAS.GetBalance operation that was already managed on Neo). It's exactly how it happens now, just giving a place to store the available GAS on operation (now it can be stored on the Transaction).

igormcoelho commented 3 years ago

Let me give a numerical example, because this reasoning you mentioned is fundamental @erikzhang .

Please inform me of the possible mistakes I'm making here!

[EDIT]: sender means signers[0] (thanks @erikzhang)

Scenario:

Current mechanism:

Proposed mechanism:

erikzhang commented 3 years ago

Why not put contractTKN as the sender?

igormcoelho commented 3 years ago

Why not put contractTKN as the sender?

I knew you were going to ask @erikzhang :joy:

[EDIT]: sender means signers[0]

Just because the real sender is Alice, and if we put ContractTKN as sender, how would it know that Alice is the "real sender" (from the co-signers list)?

erikzhang commented 3 years ago

I think sender is to pay for the fee, there is no other meaning.

igormcoelho commented 3 years ago

And don't forget about "scope"... why should ContractTKN give GAS to some TX that doesn't even execute ContractTKN?

[EDIT]: sender means signers[0]

I put ContractTKN as sender, it gives me GAS, but I execute ContractXYZ during invocation... so maybe we should first agree that "scope" is important for GAS under execution, and there could be "multiple GAS kinds" available during a single execution (some GAS is global, some GAS is for ContractTKN scope, and some other GAS is for ...).

Instead of loading ApplicationEngine with such advanced logic, I would rather put a Dictionary/Map into Transaction, and consider that execution GAS is part of the Transaction Balance, instead of considering it part of "ApplicationEngine Balance".

erikzhang commented 3 years ago

In fact there is no sender field in Transaction. We only have signers. And the sender is the first signer and pay for the fee.

why should ContractTKN give GAS to some TX that doesn't even execute ContractTKN?

You add it as one of the signers, the verify method will be invoked. You can check the transaction in verify and decide whether to pay for it.

igormcoelho commented 3 years ago

In fact there is no sender field in Transaction. We only have signers.

Ok, I'll fix description as signers[0].

You can check the transaction in verify and decide whether to pay for it.

That's the trickiest of all... there could be perhaps some theoretical way of doing this, but it's not possible "on practice" to deduce contract logic just by inspecting invocation script. It's simply so much easier to consider that, as long as signers[0] is giving GAS, at least limit that GAS into signers[0] scope. Do you agree? As long as its signature/verification is valid, you can spend its GAS.

If at least this change is made, ContractTKN logic for verify() becomes simple: it just checks that user put it as signers[0] with "an interesting" scope (itself or its own contract group).

igormcoelho commented 3 years ago

Besides the scoped GAS, which I believe now to be very important, there's another fundamental contribution of this NEP: GAS reloading during contract invocation.

As long as some contract has the ability to invoke GAS.scheduleTransfer(...) during invocation, it could load transaction with very little GAS (just to cover network fee costs) and then periodically invoke method to put more into Transaction GAS Balance. This is very good for dynamic applications that are hard to estimate on GAS costs, and it's much better than sending a lot of GAS just to get them back as refund (that could lock multiple tx invocations).

igormcoelho commented 3 years ago

I still think this NEP is valid, and honestly, I find it quite beautiful to be able to store assets on a transaction. Specially, the idea of a transaction having a temporary balance, that expires once it's finished, and then it automatically invokes then cleaning method to handle non-zero pending balances. That's why I think we should keep it, and allow non-GAS contracts to have such feature. This could allow, in some future, to have other tokens behaving in a similar manner to GAS, and like I said before, having contracts interacting with each other through Transaction Balance.

From a practical perspective on GAS, I'll open two issues directly on Neo: https://github.com/neo-project/neo/issues/2442 and https://github.com/neo-project/neo/issues/2443 For these issues, it doesn't matter if GAS is stored on ApplicationEngine or in Transaction, as on practice, every ApplicationEngine instance is bound to a distinct Transaction (for me, it's the same). So feel free to evaluate the possibility of having such features on GAS, and then we evaluate if these should apply to general assets as well.

roman-khimov commented 3 years ago

Does it makes it clearer or more confusing? :joy:

Certainly clearer, thanks @igormcoelho.

Technically I'd be concerned about these things:

Also, there is a huge difference going from

remainingFee is allocated into the following Transaction field (NEP-TTT): Transaction[GAS.id, ContractY.scope_hash] += remainingFee

to

The idea here is to create a memory map/dictionary like this (this represents the "promiseNotification" I mentioned eariler): (TransactionHash, ContractHash, Scope) -> Data

The first one tries to solve local contract sponsoring problem (and it has other solutions), the second one is much more generic and raises a question of what data and how could be stored in transaction. This generic case of this proposal probably is interesting in that it can allow limiting the amount of assets approved to use in some situations, but I'm not sure we can reliably implement this more generic case and I'm also not sure it's to be used a lot, current scoping and onNEPXXPayment mechanisms cover most of needs.

igormcoelho commented 3 years ago

Very interesting technical points @roman-khimov.

Regarding re-verification, it's not needed for the reason described in this comment here: https://github.com/neo-project/proposals/issues/137#issuecomment-823716366 On essence, precisely the same existing strategy used now for GAS Native Contract (that controls in real-time the assets allocated to fees, thus preventing re-verification) could be done using scheduleTransfer operations. They don't cause side effects "on general" during scheduleTransfer and the only contract to trace such "effects" is GAS, since it's native and it requires that to put transactions on block while covering network fees. The other tokens that implement this (except GAS) are assumed to fail during invocation, as this operation will effectively happen before the first invocation operation on contract.

Another interesting behavior of this proposal is that:

It's like creating a pool of funds that can be used during transaction execution. I think it's nice, but I agree we must find some "killer application" using this mechanism, to justify such trouble implementing it (maybe it helps on minting, maybe it helps with managing execution gas, ...).

igormcoelho commented 3 years ago

@erikzhang @roman-khimov we have a problem... (or maybe my head is not quite right at this moment) How can verify() do anything useful, if it cannot access storage during verification (or can it)? :joy: I mean, contract has an internal storage for vip customers, for token ratios, but none of this could be used during verify, is that correct?

So if it can't read storage during verification, then this NEP-TTT is absolutely necessary, in my opinion, as a way to provide accounting mechanism similar to GAS to other NEP-17 tokens, otherwise it's not possible to do contract sponsoring onchain.

I'll begin drafting a document to properly explain this to the community. If you have any questions, I'm willing to debate this as long as it is necessary, because I really think this is important for N3.

igormcoelho commented 3 years ago

I recall that the essence of this NEP-TTT is:

Now it's clear to me that this is precisely what GAS does, and that's how it manages to escape non-determinism and still avoid a second re-verification. We need that for other tokens and this NEP can provide that. If we have that, users will be able to truly enforce collateral guarantees on any NEP-17 token before Invocation happens, without breaking non-determinism and avoiding re-verification, by the virtue of fact that balances are additive (we have talked about that sooooome time ago https://github.com/neo-project/neo/issues/814). As long as it is additive, mempool can keep on memory the updated balances for any token that has GetBalance(), besides GAS, and perform these transfers for ALL transactions in block, before any invocation happens (that's the way to prevent double spending and that's exactly how GAS manages to survive it, right?). So, I really don't see much alternatives now, if we want to do this process onchain (P2P alternatives will certainly work, but I just want to give all NEP-17 tokens the same treatment that GAS currently has).

roman-khimov commented 3 years ago

How can verify() do anything useful, if it cannot access storage during verification (or can it)? :joy:

It can.

igormcoelho commented 3 years ago

It can.

It can do something useful or it can access storage? :joy: Last time I seen there was a discussion on this, but if it has access to storage, we can manage many things.

roman-khimov commented 3 years ago

It has access to storage and therefore it can do many useful things.

igormcoelho commented 3 years ago

It has access to storage and therefore it can do many useful things.

Thanks @roman-khimov , I got confused about that.

On the other hand, this means that transactions are still being re-verified (on the witness part) after every block is put... I thought that only GAS parts would require updates by now (thus preventing any re-verification). Worse, it should re-verify transactions after each transaction is executed, otherwise it risks breaking verify() sponsoring logic (as storage may have changed...). It justs not risk attacking Neo System, as GAS would be paid anyway, but from verify() perspective, it may not be a wise choice. My point here is that, if we pursue this path of forbidding storage access @erikzhang , this NEP-TTT would allow same funcionality as GAS to any NEP-17, without any re-verification after blocks or tx execution.