extensible data format for Antara modules

[Alrighttt]

@mixa84 @dimxy @ca333 @jl777 We need to design an extensible format for storing arbitrary data for arbitrary condition structures that can be stored within an OP_CHECKCRYPTOCONDTION scriptpubkey, presumably as OP_DROP data.

What I mean by this is that...

If we have the following condition structure:

{
    "type": "threshold-sha-256",
    "threshold":    1,
    "subfulfillments":  [
    {
        "type": "secp256k1-sha-256",
        "publicKey":    "02C03CDB8822DAA30241CC137FBBEE76EAD55D7F92E667DF1C8CE56E6E50824386"
    }, 
    {
        "type": "threshold-sha-256",
        "threshold":    5,
        "subfulfillments":  [
            {
                "type": "eval-sha-256",
                "code": "force_txfee(x, y)"
            },
            {
                "type": "eval-sha-256",
                "code": "must_pay_p2pk(w, x, y, z)"
            },
            {
                "type": "eval-sha-256",
                "code": "must_pay_CC(x, y, z)"
            },
            {
                "type": "eval-sha-256",
                "code": "minmax_vinsvouts(x,x,y,y)"
            },
            {
                "type": "eval-sha-256",
                "code": "minmax_CCvinsvouts(y,y,y,y)"
            }
        ]
        }
}

With the current implementation of cryptoconditions library, this structure will be hashed, but the arguments for each cannot be included in this hash.

So instead, we will remove all the arguments, hash the structure and this hash is used with OP_CHECKCRYPTOCONDITION in a scriptPubKey. Now the issue is that when this scriptPubKey needs to be spent, we know the eval codes from the scriptsig, but we do not know the arguments to provide to each.

I am proposing that we store the arguments inside the scriptPubKey via an OP_DROP(or many as each is limited to 520 bytes) appended to the end of the typical OP_CHECKCRYPTOCONDITION. Before we can begin writing modules, I believe we need to agree on a format that can handle any situation.

I am propsing we use the same BER encoding that we use for cryptocondition fullfillments and scriptsigs. As an example, the above condition structure's fullfillment(and scriptsig) would look something like this decoded(NOT EXACT)

[2] (2 elem)
  [0] (1 elem)
    [5] (2 elem)
      [0] (33 byte) 03682B255C40D0CDE8FAEE381A1A50BBB89980FF24539CB8518E294D3A63CEFE12
      [1] (64 byte) 598E1901B424A70C48047213E14BAF27DBEC0044E508CD8FE3CCC9A5C95C8D84652866…
  [1] (1 elem)
    [2] (5 elem)
      [0] (1 byte) 01
      [1] (1 byte) 02
      [2] (1 byte) 03
      [2] (1 byte) 04
      [2] (1 byte) 05

So, I am proposing we follow this same structure in order to facilitate arbitrary structures and being able to "route" arbitrary arguments to their respective eval codes.

[2] (2 elem)
  [0] (1 elem)
    [5] (2 elem) <DUMMY for secp256k1 node>
      [0] (0 byte) 
      [1] (0 byte) 
  [1] (1 elem)
    [2] (5 elem)
      [0] [1] (2 elem)
        [0] (1 byte) x
        [1] (1 byte) y
      [1] [1] (4 elem)
        [0] (1 byte) w
        [1] (1 byte) x
        [2] (1 byte) y
        [3] (1 byte) z
      [2] [1] (3 elem)
        [0] (1 byte) x
        [1] (1 byte) y
        [2] (1 byte) z
      [3] [1] (4 elem)
        [0] (1 byte) x
        [1] (1 byte) x
        [2] (1 byte) y
        [3] (1 byte) y
      [3] [1] (4 elem)
        [0] (1 byte) y
        [1] (1 byte) y
        [2] (1 byte) y
        [3] (1 byte) y

The idea is that we always use the same "index"(not sure this is the correct term) in the data structure as we do in the condition structure for the corresponding eval node. This should mean that it is extensible and no matter the condition structure, we know exactly where to look in the OP_DROP data structure for the arguments. The exact data structure can be defined by each individual module. So for example, a module might need a mask and a string to be passed to it. It could simply expect this to be passed to it and the OP_DROP data for that particular eval node would look something like:

[3] [1] (2 elem)
    [0] (1 byte) cc
    [1] (8 byte) DEADBEEF

I think it would be wise to use a well known structure such as BER as we can utilize already written libraries across many different languages.

I could use help designing this in C++, and I need someone to create helper functions.

I can design this in rust(and extend it via pyCC to python eventually).

I appreciate feedback from anyone, not just the people tagged, thanks.

[dimxy]

I dont think the threshold is a good framework to manage rules. It allows to pick M from N and this is good for mutisig but this is not the case for tx rules (as rules are not all same). And there is not always 'AND' operator suitable to apply to all rules. In real cc we usually check a property in a tx (funcid?) and apply rules according to it. Also I think the examples above are not very much realistic: we should not care about txfee amount (it should be defined by market) or how many vins/vouts a tx has. Generally, what we should care is where cc value should go, on which vouts and on which destinations. More realistic rules would be:

• the value from vins with a certain evalcode goes to vout0
• vout0 has some evalcode
• vout0 address matches the pubkey in the opreturn
• total value for cc vins with certain evalcode equals cc vouts with the same evalcode
• etc

[Alrighttt]

It allows to pick M from N and this is good for mutisig but this is not the case for tx rules (as rules are not all same). And there is not always 'AND' operator suitable to apply to all rules. In real cc we usually check a property in a tx (funcid?) and apply rules according to it.

Please give me an example of boolean logic that is not possible via thresholds and branching thresholds.

threshold 2
    threshold 1
        eval1
        eval2
    eval3

As an example, this is in practice the same as ( ( eval1 || eval 2 ) && eval3 )

Also I think the examples above are not very much realistic: we should not care about txfee amount (it should be defined by market) or how many vins/vouts a tx has.

I don't understand. Nearly every CC module we have right now cares about the number of vins and vouts in some aspect.

Also, every CC that enables a user to spend coins that don't necessarily belong to them(a faucet with a change vout back to itself for example) must enforce some kind of check on transaction fee to ensure a user cannot send an absurdly high fee to a miner.

Anyway, I think we are quite far off topic here. We know we will need to provide arguments to eval codes. We don't necessarily know what these eval codes will be, so we need an extensible format that is "future proof".

The rules themselves don't really matter in this context, just that we are able to provide arbitrary eval codes arbitrary arguments.

[dimxy]

Please give me an example of boolean logic that is not possible via thresholds and branching thresholds.

XOR and NEG logic are needed for completeness

[Alrighttt]

Would suggest opening another issue to discuss adding a new type of threshold node. Again, this is meant to discuss OP_DROP data structures.

[dimxy]

I think the format is okay to discuss after the whole concept has become clear (IMHO). The format does not affect much, could be json for PoC (and the idea looks interesting to me..)

[Alrighttt]

I will make some very basic conditions over the next couple days using JSON as the encoding for OP_DROP. I hope this will help to better convey the concept as a whole.