Add support for generating raw Omni Protocol transactions without RPC

[msgilligan]

Create library functions that can create raw Omni Protocol transactions that can be sent over the peer-to-peer network directly (e.g. via bitcoinj)

We currently have support for creating raw Bitcoin and Omni transactions using RPC calls designed for that purpose and some work towards creating raw Bitcoin transactions with bitcoinj, but do not have the capability to generate a complete Omni transaction in a client-only configuration.

[dexX7]

For what it's worth, maybe those tests can be of use:

https://github.com/dexX7/bitcoin-spock/commit/f9910cc1d6a385c07bb8cd1cd7ce77b71904ee4c

[dexX7]

I'm a bit undecided about this. I fully agree that having the ability to construct raw transactions would be very nice to have, but it needs to be decided whether the goal is to create "raw Omni transactions" (similar to builder.bitwatch.co) which could be submitted via sendrawtx_MP or go a level deeper and cover the data embedding as well, namely converting "raw Omni transactions" into obfuscated data packages and those into Bitcoin transactions.

I mention this, because #55 made me wonder, if it might be possible to use certain parts of Omni Core as external library.

In general I'm also undecided about how to handle consensus critical code. On the one hand I'd really love to refactor the current (Omni Core) code to be more modular (...), but on the other hand, and as alternative, some parts of the code could be considered as "sealed" and never to be touched again, to avoid the introduction of any unexpected behavior.

Even though it's really not rocket science, over time I observed more than once a slightly inaccurate re-implementation over at Omni Wallet, as well as some very interested, but unplanned edge case behavior. I emailed you one example. :)

So in short I suggest:

• 1. to start with constructing raw Omni transactions via builder pattern (.addPropertyIdentifier(), .addXYZ()) and cover all supported transaction types, basically picking up what was started with createPropertyHex() and createDexSellOfferHex()
• 2. evaluate external library integration (which would probably be a long shot)
• 3. (optional) clean up the core Omni Core code as far as possible
• 4. (optional) re-implement core Omni Core code, namely raw transaction construction

[msgilligan]

My goal is to "go a level deeper" and implement (1) and (4) above for creating Omni transactions and sending via bitcoinj P2P transactions. Top priority is Tx 0 (Simple Send). I'd prefer a pure Java implementation, but would be open to using (2) if it were available.

[dexX7]

Let's first clarify something else: I assume this is still about class B/bare multisig transactions, and it should not wait until transactions are created with OP_RETURN, which could take a while, right?

I'll focus on schematics of the transaction encoding, because the builder shouldn't be too difficult. I finished that part in a seperated C++ version yesterday and I'm sort of satisfied with the results, so we could basically convert this into Groovy or Java. There is some overhead for speed optimizations and a Groovy version is probably going to be shorter.

These are basically the building blocks, operating on bytes (shown as hex in the examples):

1) AddSequenceNumbers() + Unit tests: After every 30th byte an ascending number, starting from 1, but no more than 255, is inserted:

// before:
00000000000000000000000000000000000000000000000000000000000000..

// after:
010000000000000000000000000000000000000000000000000000000000000200..

2) ObfuscateUpperSha256() [Sha256(), UpperSha256(), XorHashMix()] + Unit tests: Given is a "stream" of bytes and a "seed", which is the Bitcoin address of the sender as string:

• data: the bytes are handled as chucks with a length of 31 byte
• hash: the seed as string is hashed via SHA256
• hash: the result is converted to upper characters
• hash: the result becomes the next seed, and is returned
• the data chuck and this hash are xor'ed on a byte level
• the xor result (which has a length of 32 byte) is trimmed to 31 byte
• the process repeats until the end of the "stream" is reached, but no more than 255 times

Hashing (without trimming):

// seed:
1CdighsfdfRcj4ytQSskZgQXbUEamuMUNF
// sha256 + convert to upper, round 1:
1D9A3DE5C2E22BF89A1E41E6FEDAB54582F8A0C3AE14394A59366293DD130C59
// sha256 + convert to upper, round 2:
0800ED44F1300FB3A5980ECFA8924FEDB2D5FDBEF8B21BBA6526B4FD5F9C167C
// sha256 + convert to upper, round 3:
7110A59D22D5AF6A34B7A196DAE7CCC0F27354B34E257832B9955611A9D79B06

Combined with xor:

// seed:
1CdighsfdfRcj4ytQSskZgQXbUEamuMUNF
// data blob:
0100000000000000010000000002faf080
// first upper hash:
1D9A3DE5C2E22BF89A1E41E6FEDAB54582F8A0C3AE14394A59366293DD130C59
// blob ^ hash round 1:
1c9a3de5c2e22bf89b1e41e6fed84fb502f8a0c3ae14394a59366293dd130c59
// trimmed to 31 byte:
1c9a3de5c2e22bf89b1e41e6fed84fb502f8a0c3ae14394a59366293dd130c

3) ConvertToPubKeys() [CreatePubKey(), ModifyEcdsaPoint()] + (very few) Unit tests: A "stream" of bytes is converted into pubkey keys:

• the bytes are handled as chucks with a length of 31 byte at most
• each chuck is prefixed with 0x02 or 0x03
• each chuck is resized or padded to 33 byte
• each chuck is used to create a public key
• the last byte of each public key is modified until the public key is valid

// invalid public key, which does not represent a valid ECDSA point:
02777ac9576cb08fb869efd4be2ca094c55808054e2220285f3c754d59361b3007
// valid public key after modifying the last byte:
02777ac9576cb08fb869efd4be2ca094c55808054e2220285f3c754d59361b300c

// some blob of data:
123456
// converted into a valid public key, with public key prefix and modified last byte:
031234560000000000000000000000000000000000000000000000000000000001

4) EncodeBareMultisig(): Given is a "stream" of bytes and a public key of the sender, which should be used to redeem the transactions at some point, to avoid the creation of dust:

• the bytes are transformed into public keys (step 3)
• packets of m-of-n bare multisig scripts are created
• the first public key used in each m-of-n packet is the one from the sender
• it's either 1-of-2 multisig or 1-of-3
• not strictly required for this step, but I also create outputs from those packets

// 1-of-2 bare multisig script:
OP_1 [redeeing public key] [data public key 1] OP_2 OP_CHECKMULTISIG
// 1-of-3 bare multisig script:
OP_1 [redeeing public key] [data public key 1] [data public key 2] OP_3 OP_CHECKMULTISIG

5) EncodeBareMultisigObfuscated(): Given is a "stream" of bytes and a public key of the sender, and when combining all the steps above, "Class B encoded outputs" can be created:

• insert sequence numbers into the stream of bytes (step 1)
• obfuscate the data by hashing and xoring (step 2)
• create public keys from that result (step 3)
• embed the public keys into bare multisig outputs (step 4)

Input:

// the address of the sender, used as seed for hashing:
mvayzbj425X55kRLLPQiuCXWUED6LMP65C
// public key of the sender, used to redeem dust later:
0347d08029b5cbc934f6079b650c50718eab5a56d51cf6b742ec9f865a41fcfca3
// payload (it's a "create property" transaction):
00000032010001000000000000426172654d756c7469736967546f6b656e73006275696c6465722e62697477617463682e636f000000000000000f4240

Result (via RPC call obfuscated_multisig):

{
  "source": "mvayzbj425X55kRLLPQiuCXWUED6LMP65C",
  "redeemer": "0347d08029b5cbc934f6079b650c50718eab5a56d51cf6b742ec9f865a41fcfca3",
  "payload": "00000032010001000000000000426172654d756c7469736967546f6b656e73006275696c6465722e62697477617463682e636f000000000000000f4240",
  "txouts": [{
    "value": 0.00000786,
    "asm": "1 0347d08029b5cbc934f6079b650c50718eab5a56d51cf6b742ec9f865a41fcfca3 03e2e98198f331c436644f88b5a6bc5c65df64d53457d624ed05e78dba40dd5e01 02fac1e512bac2575554a5dee8a345fc773615af68a09d0291473316fe39087e06 3 OP_CHECKMULTISIG",
    "hex": "51210347d08029b5cbc934f6079b650c50718eab5a56d51cf6b742ec9f865a41fcfca32103e2e98198f331c436644f88b5a6bc5c65df64d53457d624ed05e78dba40dd5e012102fac1e512bac2575554a5dee8a345fc773615af68a09d0291473316fe39087e0653ae"
  }, {
    "value": 0.00000684,
    "asm": "1 0347d08029b5cbc934f6079b650c50718eab5a56d51cf6b742ec9f865a41fcfca3 03fabf5862d9719cf2fc07b1c1a2204cb4d74ea3e72f358f31f32f90c4629c2100 2 OP_CHECKMULTISIG",
    "hex": "51210347d08029b5cbc934f6079b650c50718eab5a56d51cf6b742ec9f865a41fcfca32103fabf5862d9719cf2fc07b1c1a2204cb4d74ea3e72f358f31f32f90c4629c210052ae"
  }]
}

6) Create actual transactions:

• use the Omni transaction builder to create a payload (similar to builder.bitwatch.co)
• convert that payload into class B encoded transaction outputs (step 5)
• create a new (raw) transaction
• add an output to 1EXoDusjGwvnjZUyKkxZ4UHEf77z6A5S4P or mpexoDuSkGGqvqrkrjiFng38QPkJQVFyqv
• add one or more data carrying bare multisig outputs, but no more than 255
• optionally add a change output

---

It probably seems unnecessary to take a blob of data, slice it into chucks, do some transformation, convert it into a stream, then slice it again into chucks, ... but my intention was to abstract data from the actual data embedding, because m-of-n "packets" are rather specific, whereby other embedding mechanisms may not operate on "public keys" for example.

This was intended as overview and introduction, so you get a feeling for what's happening under the hood, and I'm not sure, if this is a great solution, but either way, I suggest to resolve this issue in a test driven manner, namely by creating specifications and tests first, then start the implementation of the actual methods.

Please let me know, what you think. :)

[msgilligan]

Thanks for this, @dexX7! While I would love to skip straight to Class C, there is a need to create Omni transactions on Android prior to the release of Omni Core with Class C support.

I want to code the implementation using Java 7 syntax with Java 6 compatible library usage, so it can run on Android. We can write Spock tests in Groovy, and although Groovy now works on Android, I'd like to keep the core implementation as generic as possible.

This approach looks great and I'll start writing Java code and Spock tests based on this later today!

[dexX7]

Awesome. I'd love a Java implementation and I'm very curious about the results. :) Please let me know, if you need any help or clarifications.

This is sort of a different topic, but since you mentioned an Android wallet a few times: is there already a goal/plan? I'm especially wondering, where the actual data is coming from, e.g. the balances, as this is tackles only the "encoding", but not "decoding" side.

[msgilligan]

For now I'm focused on the encoding/sending side rather than the balance-calculation side. I'm planning to integrate with an open source Android wallet that has not been released yet. For this initial version balances will have to come from a server API -- though I'm not sure which API that will be. Possibly the Omniwallet API.

[msgilligan]

Step 1, "sequence numbers" committed: 48a1fdd9d1bc3c47bd9239bc90690d9120ae52b9 (tests in previous commit)

Thanks, again @dexX7

[msgilligan]

Step 2, "obfuscation" committed: 72c797368da78fcf10a70556783673782c5a297b

(This is work-in-progress and I expect some code cleanup after everything is working)

[dexX7]

I took a look at BitcoinJ to get an idea how the rest could be done.

For CreatePubKey() and ModifyEcdsaPoint():

CPubKey pubKey;
pubKey.Set(vchFakeKey.begin(), vchFakeKey.end());

... there is:

// in package org.bitcoinj.core
ECKey ECKey.fromPublicOnly(byte[] pub)

... which could be used instead.

Although the generation would have do be done slightly different, because BitcoinJ, in contrast to Bitcoin Core, does not allow the creation of invalid or empty public keys.

CreatePubKey() and ModifyEcdsaPoint() would instead operate on the byte[] array and it's expected that ECKey.fromPublicOnly() throws java.lang.IllegalArgumentException: Invalid point compression, if the fake public key is not fully valid. The last byte would be modified, until no exception is thrown (and the generated key is valid).

---

In EncodeBareMultisig():

CScript script = GetScriptForMultisig(1, keys);

... could be replaced by:

// in package org.bitcoinj.script
Script createMultiSigOutputScript(int threshold, List<ECKey> pubkeys)

... where threshold = 1, because we want to generate 1-of-2 or 1-of-3 bare multisig scripts.

---

There also seems to be a class for transaction outputs, namely org.bitcoinj.core.TransactionOutput.

[msgilligan]

Thanks for this. I'm hoping to make a stab at this over the weekend.

[msgilligan]

@dexX7 - I didn't specify which weekend now did I? ;)

[msgilligan]

First (rough) cut at Step 3): f1b4cbe1006bf269645bba261b8cbdc25412900c

[dexX7]

@msgilligan: Hehe.. :)

I adopted what you mentioned regarding the obfuscation and removed the padding from the upper SHA256 modification. This introduces a semi-issue though:

A public key with a size of 33 byte has 1 byte prefix and 1 byte ECDSA modification byte, but with less than 32 byte "pubkey-payload", the last bytes would be 0, or the ECDSA byte would always start from 0. As result, the only the first half of final packets/payloads would look random. To overcome this, I basically now do:

• After adding sequence numbers, and before doing the obfuscation, the payload is padded to to a size of n * 31 byte.
• During the public key creation, and if the payload has a size of less than 32 byte, the last byte pubkey[32] is chosen randomly, to have a random seed for the ECDSA modification.

It might not be 100 % intuitive, and ConvertToPubKeys() still slices into 31 byte packets, but CreatePubKey() can work with 1-31 byte (pubkey padded with 0, last byte random), as well with 32 (no byte modified or padded before ECDSA modification).

The goal was to be able to create unobfuscated pubkeys with random ECDSA seed, and to be able to have fully obfuscated pubkeys without trailing zeros, also with random ECDSA seed.

Hope this makes sense. :) The related commits are:

• No padding of xor-result during obfuscation
• Improve payload to public key conversion
• Explicitly resize the payload before obfuscation + Move PadBeforeObfuscationIn into new file

A bunch of new unit tests for public key conversion are available.

[msgilligan]

Thanks, @dexX7. I'm going to push ahead with steps 4, 5, and 6 and then revisit your above comments. I'd really like to get proof-of-concept end-to-end Tx creation coded and maybe even running and then come back to some of the finer points. Does that sound like a reasonable approach for me?

[dexX7]

@msgilligan: yes, sounds fine! :) And from what I can see: you're actually pretty close to a POC.

[msgilligan]

I have rough cut of the txoutput/script encoding checked in as WIP: ba09592408c300460f7e730fc3a4970f9823e9f7

[msgilligan]

I've moved some code from the TestSupport class to two new classes:

1. RawTxBuilder: a Java class in omnij-core to build transactions in Hex strings. Make sure to see the RawTxBuilderSpec for sample code.
2. ExtendedTransactions: a Groovy trait (mix-in) that uses RawTxBuilder and sendrawtx_MP to create and send Omni transactions not supported by existing RPCs. It lives in the omnij-rpc module.

@dexX7 , have you seen this yet?

[dexX7]

Yep, saw it. :) So what's up next? I'm still struggleing a bit with the general design, and I really like your class based approach. Roughly, to construct transactions, it looks like the sequence number insertion, the obfuscation, as well as the public key conversion are done. So actually, only an Exodus output has to be added, and everything converted into transactions, as far as I can see?

createRawTransaction() might be used and from:

String createRawTransaction(Address fromAddress, Map<Address, BigDecimal> outputs)

... turned into something like:

Transaction createTransactionClassB(Address fromAddress, Map<Coin, Script> outputs)

... or:

Transaction createTransactionClassB(Address fromAddress, List<TransactionOutput> outputs)

[msgilligan]

Yes, I'm currently working on code that adds an Exodus output and reference address output, but don't have it quite working yet. When I have it working, I'll check it in. After that I want to try to factor the code to a more idiomatic, modern Java style -- perhaps with a true "builder" pattern for creating the byte[] data and with an API that should work for both Class B and Class C transactions (when supported.)

[dexX7]

Class C is going to be easier, as it is not obfuscated and just plain data basically.

I want to try to factor the code to a more idiomatic ...

This is something I'm really interested in, and would love to learn from it. :) If there are specific issues regarding the embedding or transaction construction, please feel free to ping me.

[msgilligan]

As of commit bb4024f051a0a4fe8a675606d8272e4405854447, the OmniTxBuilder class can create signed Omni transactions. More work is needed, but basics transaction creation seems to be working correctly in the OmniTxBuilderIntegSpec integration test.

@dexX7 thanks for all your help!

[dexX7]

Wooho, great progress! :) And with https://github.com/OmniLayer/omnicore/pull/13 and the raw transaction dumping, a lot of new test data can be generated to further push the data embedding and obfuscation. :)

[dexX7]

Are you also interested in decoding transactions? It's pretty straight forward for the most part, and simply a reversal of the encoding. The most complex part is probably the "sender selection" (which serves as seed for the deobfuscation), which I currently dislike due to it's complexity, it's unexpected bug-ish behavior and the requirement to fetch multiple transactions.

That aside, there are some further constraints, namely only some specific transaction output types are allowed as input, and others would invalidate the whole transaction, but I started with the core algorithm and have a bunch of unit tests ready:

https://github.com/dexX7/bitcoin/blob/aecc38ce396bcd15abf71e6ed96f6d2fae90c669/src/extensions/core/transactions.cpp#L37-L129 https://github.com/dexX7/bitcoin/blob/aecc38ce396bcd15abf71e6ed96f6d2fae90c669/src/extensions/test/sender_selection_tests.cpp

[msgilligan]

I am interested in decoding transactions, but it's not my current priority. My current priority is to get transaction creation, signing, and sending (via P2P or via a server) on an Android device.

[msgilligan]

I think we can close this issue now. There's more work to be done, but the basics are there. We can open new issues for specific tasks. What do you think, @dexX7 ?

[dexX7]

Your call.

But given that the initial goal was met, I think this can be closed. If there is need for a follow up, a new thread can be created.

[msgilligan]

This was the best thread ever! Thanks for all the great comments, @dexX7 !