Cannot reconstruct transactions referencing contracts in the same transaction via the Ledger API

[cocreature]

For things like Daml Script dumps, you want to be able to turn the root commands of a transaction tree into Ledger API commands. This seems easy at first, create -> create command, exercise -> exercise command but it runs into issues once one command references an earlier contract created in the same transaction. You can in some cases translate this into createAndExercise but this falls apart if the exercise does not follow the create immediately. Here is a concrete example. Bob does not see the exercise but he sees cid <- create, cid2 <- create, exercise cid … which is not possible via the ledger API directly.

I see two options for Daml Script dumpsd

1. Just error out, clearly not a great solutoin.
2. We can generate a helper template and call createAndExercise on that but that doesn’t produce the original transaction structure so that’s kind of crappy and it’s also rather confusing.

I don’t really like either option but 2 seems slightly better.

module Main where

import Daml.Script

template T1
  with
    p : Party
  where
    signatory p
    choice Create : ContractId T2
      with
        p' : Party
      controller p
      do cid <- create T2 with p1 = p, p2 = p'
         create T2 with p1 = p, p2 = p'
         exercise cid C

template T2
  with
    p1 : Party
    p2 : Party
  where
    signatory p1
    observer p2
    choice C : ContractId T2
      controller p1
      do create this

test = do
  let Some alice = partyFromText "Alice"
  let Some bob = partyFromText "Bob"
  cid <- submit alice $ createCmd (T1 alice)
  submit alice $ exerciseCmd cid (Create bob)

[cocreature]

Two more ideas proposed by @bame-da:

1. Give up transactionality and use separate transactions (only where we have to). I think this might be a pretty reasonable option but it probably should be opt-in. We could at least add comments in the daml script to indicate grouping.
2. Medium to long-term @bame-da has a proposal to add "custom commands" to the ledger. This is similar to the createAndExercise solution, you would also have to deploy those commands (although you could potentially do it in one API call) but it avoids the two extra nodes.

[aherrmann-da]

Note that Daml script dumps only reconstruct the root events of a transaction tree into Daml script commands. I.e. the example in the issue description above results in the following Daml script dump:

dump : Parties -> Script ()
dump Parties{..} = do
  t1_1_0 <- submit alice_0 do
    createCmd Main.T1 with
      p = alice_0
  tree <- submitTree alice_0 do
    exerciseCmd t1_1_0 Main.Create with
      p$u0027 = bob_0
  pure ()

In this case the ordering of creates and exercises in child events do not pose an issue and the generated Daml script dump correctly reproduces the history. (Apart from the bug in decoding p' as p$u0027)

However, the issue is valid. Using exerciseByKeyCmd it is possible to construct the ordering issue in the root events. Take the following Daml script:

{-# LANGUAGE ApplicativeDo #-}

module Main where

import Daml.Script

template ContractA
  with
    owner : Party
  where
    signatory owner
    key owner : Party
    maintainer key
    controller owner can
      ArchiveB : ()
        do
          Some b <- lookupByKey @ContractB owner
          archive b

template ContractB
  with
    owner : Party
  where
    signatory owner
    key owner : Party
    maintainer key

setup : Script ()
setup = script do
  alice <- allocatePartyWithHint "Alice" (PartyIdHint "Alice")

  submit alice do
    _ <- createCmd ContractA with owner = alice
    _ <- createCmd ContractB with owner = alice
    exerciseByKeyCmd @ContractA alice ArchiveB
    pure ()

  pure ()

Note that the ArchiveB choice requires ContractB to be instantiated. However, Daml script dump as of https://github.com/digital-asset/daml/pull/9150#discussion_r595366589 will incorrectly reconstruct this as a createAndExercise as follows:

dump : Parties -> Script ()
dump Parties{..} = do
  tree <- submitTree alice_0 do
    createAndExerciseCmd
      Main.ContractA with
        owner = alice_0
      Main.ArchiveB
    createCmd Main.ContractB with
      owner = alice_0
  pure ()

This will fail as lookupByKey @ContractB owner will return None.

[cocreature]

Note that Daml script dumps only reconstruct the root events of a transaction tree into Daml script commands.

They reconstruct the roots that you see. Bob does not see the exercise as the root, they only see the exercise, create & create as roots.

[aherrmann-da]

Ah, I see, yes using only Bob produces the following dump

dump : Parties -> Script ()
dump Parties{..} = do
  tree <- submitTree alice_0 do
    createAndExerciseCmd
      Main.T2 with
        p1 = alice_0
        p2 = bob_0
      Main.C
    createCmd Main.T2 with
      p1 = alice_0
      p2 = bob_0
  pure ()