[Reputation Oracle] Refactor `processPaidWebhooks` method

[Dzeranov]

Description Rework webhook processing by introducing new entities for better tracking and error handling. This includes adding a WebhookOutgoingEntity to manage outgoing webhooks, an EscrowCompletionTrackingEntity to track escrow status, and enhancing error handling mechanisms to prevent redundant operations and ensure failed webhooks are properly retried.

Motivation With current implementation in case escrow has been successfully completed but webhook wasn't sent, cron job will be launched again trying to update escrow status to completed and this operation will fail causing webhook to be stuck.

Implementation details:

• [x] Update WebhookIncomingEntity model:

  • Remove result_url.
  • Remove PAID status.
  • Add failedReason.
  • Add retriesCount.

• [x] Add WebhookOutgoingEntity model:

  • Add id, url, payload (json), hash (url, payload), status (pending, sent), failedReason, retriesCount.

• [x] Add EscrowCompletionTrakingEntity model:

  • Add id, chain_id, escrow_address, final_results_url, status (pending, paid, completed).

• [x] Cron Module to track incoming webhooks:

  • Monitor WebhookIncomingEntity entries with status pending.
  • Create a new record in EscrowCompletionTrakingEntity with status pending.
  • Update the status of the WebhookIncomingEntity to completed.

• [x] Cron Module to track escrow completion records:

  • Monitor EscrowCompletionTrakingEntity entries with status paid.
  • Check the current escrow status:
    • If escrow is Paid, invoke escrowClient.complete().
    • If escrow is Partial/Completed, skip completion.
  • Create WebhookOutgoingEntity with status pending
  • Update the status of the EscrowCompletionTrakingEntity to completed.

• [x] Cron Module to track escrow completion records:

  • Monitor EscrowCompletionTrakingEntity entries with status pending.
  • Check the current escrow status:
    • If escrow is Launched, invoke escrowClient.complete().
    • Check if result_url exists.
      • If empty then save in result_url, execute payouts.
      • if exists then execute payouts.
    • If escrow is Partial, skip payouts.
  • Update the status of the EscrowCompletionTrakingEntity to paid.

• [x] Cron Module to track outgoing webhooks:

  • Monitor WebhookOutgoingEntity entries with status pending.
  • signAndSend
  • Update the status of the WebhookIncomingEntity to sent.

• [x] Update migrations

• [x] Update unit tests

[eugenvoronov]

Description Need to rework processPaidWebhooks cron job. Desired behaviour: In case await escrowClient.complete() is successful we don't need to try complete the escrow again. In case webhookService.sendWebhook() fails, we want to launch a cron job only for resending failed webhook(s).

My suggestion here is to implement some kind of a webhook queue because we are sending webhooks to multiple instances (JL, ExO, RecO) and we want to track which notification failed to be sent.

Motivation With current implementation in case escrow has been successfully completed but webhook wasn't sent, cron job will be launched again trying to update escrow status to completed and this operation will fail causing webhook to be stuck.

Implemented:

• [x] Update WebhookIncomingEntity model:

  • Rename WebhookIncomingEntity to WebhookEntity.
  • Add an enum field type to WebhookEntity with values: in and out.
  • Add a failedReason field to store the reason for failed webhooks.
  • Add a url field to store the webhook url.

• [x] Cron Module to track in webhooks:

  • Monitor WebhookEntity entries with type: in and status pending.
  • For each webhook, check the current escrow status:
    • If escrow is incomplete, invoke escrowClient.complete().
    • If escrow fails, increment retriesCount and retry the operation.

• [x] Create webhooks:

  • Generate three WebhookEntity instances with type: out and status pending.

• [x] Update in webhook status:

  • Update the status of the WebhookEntity with type: in to completed.

• [x] Cron module to track out webhooks:

  • Monitor WebhookEntity entries with type: out and status pending.
  • For each out webhook:
    • Use webhookService.sendWebhook() to send the notification.
    • If sending fails, increment retriesCount and store the failedReason.
    • Upon successful delivery, update the related in webhook status to completed.

• [x] Update migrations

• [x] Update unit tests