Add Commit Hooks to write extra information within the same transaction

[osopardo1]

Table Formats encapsulate write actions into an Optimistic Transaction. Various processes could try to commit the info to the Transaction Log, but only one would succeed, making the others retry the operation.

When the user wants to write any extra metadata about the process or to do any checks on the data, it has to wait until the transaction is successfully over, thus leading to delays and possible inconsistencies.

The idea is to add commit hooks that allow to run custom code before or after calling the Delta Lake txn.commit() method.

[osopardo1]

If you could provide any other necessary info regarding this enhancement would be great @Jiaweihu08 @cugni

[osopardo1]

More details about the implementation in #319

The idea is to have an interface such as:

/**
 * A hook that can be run before a commit
 */
trait QbeastPreCommitHook {

  /**
   * The name of the hook
   */
  val name: String

  type PreCommitHookResponse = Map[String, String]

  /**
   * Run the hook with the given actions
   *
   * @param actions
   *   the actions to run the hook with
   */
  def run(actions: Seq[Action]): PreCommitHookResponse
}

Users can extend that interface and add their code to run pre-commit checks or actions. Those actions need to return a Map[String, String] of information they want to write on the Commit Log about the execution of the hook.

Registering a PreCommitHook

In terms of usage from the DataFrame API, one user may call the hook as:

df.write
.format("qbeast")
.option("qbeast.preCommitHook.my_hook_1", "my.hook.class.path")
.option("qbeast.preCommitHook.my_hook_1.arg1", "first_argument")
.save(...)

In that way, we will identify each hook by its name on the options.

Output of a PreCommitHook

Each implementation of a PreCommitHook must return a Map[String, String] with any useful information to register in the Commit Log. Qbeast will add the prefix registered in the options to the result map keys, and it will save it on the tags variable in the CommitInfo.

"commitInfo" {
    "tags" { 
          "qbeast.preCommitHook.my_hook_1.write_id":"id",
          "qbeast.preCommitHook.my_hook_1.cpu_time":"30ms"
     }
 }