Daml Scenario service crashes when the output transaction is too big

[remyhaemmerle-da]

Affected Daml version 2.8.3

Bug description

The scenario service crashed with an usefulness error "Scenario service backend error: BErrorFail StatusCancelled" when running a scenario producing a big transaction.

To reproduce

The following script crashes when run from Daml studio

module Main where

import Daml.Script

template T
  with
    p  : Party
  where
    signatory p
    nonconsuming choice Flood : ()
        with n : Int
      controller p
      do
        if n == 0 then
          pure ()
        else do
          createAndExercise (T p) (Flood $ n - 1)
          createAndExercise (T p) (Flood $ n - 1)
          pure ()

setup = script do
  alice <- allocatePartyWithHint "Alice" (PartyIdHint "Alice")
  aliceId <- validateUserId "alice"
  createUser (User aliceId (Some alice)) [CanActAs alice]

  cid <- submit alice do createCmd T with p = alice

  submit alice $ exerciseCmd cid (Flood 12)

  pure ()

Expected behavior

We should report a more explicit message.

Additional context

I believe it is the root cause of the issue reported on the daml forum

[basvangijzel-DA]

@samuel-williams-da This might tie in well with the overall error piping approach.