Code Sample of how to execute orders, merklize the orders, publish the hash on chain.

[clangenb]

Preliminaries:

• [x] Rebase #4 on to https://github.com/integritee-network/worker/tree/sdk-v0.11.0-polkadot-v0.9.37. This is one of our release branches, where we ensure that it stable and reduce the likelihood that it breaks with cargo update. This rebase will also give you access to the new publish_hash transaction that you need for publishing the hash on the integritee parachain, see example of how it is used no in the shield_funds trusted call: https://github.com/integritee-network/worker/pull/1174.

In the trusted calls, I would expect something like this:

TrustedCall::pay_as_bid(_who, orders_file) => {

      let raw_orders = fs::read_to_string(orders_file).expect("error reading file");
      let orders: Vec<Order> = serde_json::from_str(&raw_orders).expect("error serializing to JSON");
      let market_input = MarketInput { orders };

      // You don't need to call `collect()` on the iterator, as the `merkle_root` accepts an iterator.
      // This saves us `O(n)` computation time.
      let order_merkle_root = merkle_root::<Keccak256, _>(orders.iter().map(|o| o.encode()));
      let pay_as_bid: MarketOutput = pay_as_bid_matching(&market_input);

      // Store current market output/hash in the state in the state, so you don't have to recalculate it in the getters. (If this is needed).
      sp_io::set(b"MarketOutput", pay_as_bid.encode());
      sp_io::set(b"OrdersMerkleRoot", pay_as_bid.encode());

      let elapsed = now.elapsed();
      info!("Time Elapsed for PayAsBid Algorithm is: {:.2?}", elapsed);

      // Send proof of execution on chain.
      // calls is in the scope from the outside
      calls.push(OpaqueCall::from_tuple(&(
          node_metadata_repo.get_from_metadata(|m| m.publish_hash_call_indexes())??,
          order_merkle_root ,
          Vec::<itp_types::H256>::new(), // you can ignore this for now. Clients could subscribe to the hashes here to be notified when a new hash is published.
          b"Published merkle root of an order!".to_vec(),
      )));
      Ok(())
},

in the trusted Getters something like this:

TrustedGetter::pay_as_bid_proof(who, orders_file, leaf_index) => {

      let raw_orders = fs::read_to_string(orders_file).expect("error reading file");
      let orders: Vec<Order> = serde_json::from_str(&raw_orders).expect("error serializing to JSON");

      let merkle_proof =
          merkle_proof::<Keccak256, _, _>(orders.iter().map(|o| o.encode()), leaf_index.into());

      Ok(Some(merkle_proof))
},

For the verification, I am unsure if I understand your approach, was this just a test? I would assume that you don't need to go into the enclave to verify the proof, I would just do a regular CLI where you can supply the merkle proof and the client verifies it for you:

/// Take this with a grain of salt, I this is more of a brain dump, but the final solution will look very similar to this one I suppose.
#[derive(Parser)]
pub struct VerifyMerkleProofCommand {
    /// Well-known merkle root, which the clients knows from either the parachain or because it has been printed in the previous commands.
        /// It is probably the easiest to supply this as a hex-string: `0x....`
    MerkleRoot: String,

        /// Merkle Proof
        /// Here I am unsure myself, the easiest is probably to supply the json of the merkle proof that has been debug printed
       /// in the CLI in the `pay_as_bid_proof` cmd, like: { proof: ..., leaf: .. }/
    MerkleProof: String,

        leaf_index: u32,
        /// To be honest, I don't understand this param of the proof verification.
        orders_encoded_len: u32
}

[m-yahya]

@clangenb I'm unable to resolve conflicts for lock files. These conflicts appear again and again. is there a way to skip conflicts for lock files?

[clangenb]

Yeah lock files are always a hassle, usually I just accept the other file, and regenerate the lock file after the merge/rebase process.

[GodwinOkwuibe]

There is no need to go through the enclave for hash verification as mentioned by @clangenb

[m-yahya]

I'm rebasing in the following way:

// step 1
git remote add -f sdk-v0.11.0-polkadot-v0.9.37 https://github.com/integritee-network/worker.git

// step 2
git rebase cI/simplyR remotes/sdk-v0.11.0-polkadot-v0.9.37/sdk-v0.11.0-polkadot-v0.9.37

I'll accept the incoming changes for lock files for now. and will regenerate afterward. is that fine? @clangenb

[m-yahya]

after rebasing, cargo clippy works fine. However, the make command gives following error:

error: linking with `cc` failed: exit status: 1

The error is the same even after cargo -update. any help how to fix it?

[clangenb]

I suggest this is because you also need to use a new docker image: https://hub.docker.com/r/integritee/integritee-dev/tags if your version is <v0.1.11, this is the reason. Otherwise, I need more logging output.

[m-yahya]

I could finally compile it with v0.1.12.

[m-yahya]

@clangenb can you please check this commit for the implementation of pay_as_bid algorithm? As we are not returning anything from trusted_call, the respective cli is just executing the algorithm. I'm converting orders to a String vector before encoding them. This is because let orders_encoded: Vec<Vec<u8>> = orders.into_iter().map(|o| o.encode()).collect(); gives the error method cannot be called onsimplyr_lib::Orderdue to unsatisfied trait bounds. To fix the above error, I have derived Decode to Orderstruct and OrderType enum. But this results in the following errors:

// line 53
pub cluster_index: Option<usize>,
// error
the trait `parity_scale_codec::Decode` is not implemented for `core::option::Option<usize>`

However, with the String vector method the algorithm seems working.

Please also take a look at this commit for the implementation of pay_as_bid_proof, it is in the error stage because of the mismatched types in the return statement:

fn execute(self) -> Option<Vec<u8>> {
expected `std::option::Option<std::vec::Vec<u8>>` because of return type

 Ok(Some(merkle_proof))
 ^^^^^^^^^^^^^^^^^^^^^^ expected enum `std::option::Option`, found enum `std::result::Result`
= note: expected enum `std::option::Option<std::vec::Vec<_>>`
               found enum `std::result::Result<std::option::Option<MerkleProof<H256, std::vec::Vec<_>>>, _>`

I tried to decode and unwrap the proof, but it did not work. Can you please guide me a little bit here? thanks

[clangenb]

1. Yeah, usize is the only type that is incompatible with scale-codec because it is of variable size dependeing on the architecture and as the codec is not self-descriptive, it can't contain data about it's length (except for dynamically sized types like vec, there the length is part of the encoding). . You need to use u32, or u64 instead.

2. Just remove the wrapping in Ok() at the end.

[m-yahya]

1. I'll discuss this with Timo if can use u32 or u64 instead.

2. I already have tried removing Ok wrapper but still getting the mismatched types error:

   error[E0308]: mismatched types
   --> app-libs/stf/src/getter.rs:189:11
   |
   189 |                     Some(merkle_proof)
   |                     ---- ^^^^^^^^^^^^ expected struct `std::vec::Vec`, found struct `binary_merkle_tree::MerkleProof`
   |                     |
   |                     arguments to this enum variant are incorrect
   |
   = note: expected struct `std::vec::Vec<_>`
              found struct `binary_merkle_tree::MerkleProof<sp_core::H256, std::vec::Vec<_>>`

[clangenb]

Ah sorry I did not catch this one, yeah this is because that the merkle proof needs to be encoded, sadly the merkle proof structure does not implement codec because it contains usize. So what I suggest here, is that we have to redefine the MerkleProof in our crate:

/// Custom Merkle proof that implements codec
///
/// The difference to the original one is that is implements the scale-codec and that the fields contain u32 instead of usize.
#[derive(Debug, PartialEq, Eq, Decode, Encode)]
pub struct MerkleProofWithCodec<H, L> {
    /// Root hash of generated merkle tree.
    pub root: H,
    /// Proof items (does not contain the leaf hash, nor the root obviously).
    ///
    /// This vec contains all inner node hashes necessary to reconstruct the root hash given the
    /// leaf hash.
    pub proof: Vec<H>,
    /// Number of leaves in the original tree.
    ///
    /// This is needed to detect a case where we have an odd number of leaves that "get promoted"
    /// to upper layers.
    pub number_of_leaves: u32,
    /// Index of the leaf the proof is for (0-based).
    pub leaf_index: u32,
    /// Leaf content.
    pub leaf: L,
}

/// Then we can also implement conversion of the two types to make the handling more ergonomic:

impl<H, L> From<MerkleProof<H, L> for  MerkleProofWithCodec<H, L> {
     fn from(source: MerkleProof<H, L>) -> Self {
       Self {
               root: source.root,
               proof: source.proof,
               number_of_leaves: source.number_of_leaves.try_into().expec("We don't have more than u32::MAX leaves; qed"),
               leaf_index: source.leave_index.try_into().expect("Leave index is never bigger than U32::Max; qed");
               leaf: source.leaf
       }
    }

}

In your getter you can do the following:

let proof: MerkleProofWithCodec<_, _> = merkle_proof<Keccak256, _, _>::(data).into();

// return the proof as follows:
Some(proof.encode())

[clangenb]

note: the above might contain some typos, it has not been tested.

[m-yahya]

@clangenb can you please take a look at this commit for the implementation of pay_as_bid_proof method?

[clangenb]

I would say this is done