Quoting and Transfer Services do not apply Canonical JSON transformation as part of the Duplicate Check logic

[mdebarros]

Request Summary:

Quoting and Transfer Services do not apply Canonical JSON transformation as part of the Duplicate Check logic.

This means the current Duplicate Check logic requires the payload structure to be exactly the same for a Duplicate to be found. So if a Duplicate request has the same contents, but the JSON structure has changed (i.e. key ordering), the Duplicate check will fail.

I propose that we make the Canonical JSON (ref: https://datatracker.ietf.org/doc/html/rfc8785) transformation optional as part of the Duplicate Check Logic.

Current implementation options (in order of preference):

1. Add a new column to the Duplicate Table on Quotes/Transfers which includes a versioning/naming of the hash transform used to calculate the Duplicate Check. If the value is null/undefined then the standard/legacy Duplicate Check logic would be applied. In this option, the decision on how to calculate the hash will be backward compatible and completely database-driven. All duplicate hash operations going forward would then store the hash transform used in the new field, which would be applied if and only if a duplicate record was identified by the transfer/quote IDs. <--I initially thought that this option could cause a performance issue, but after re-looking at the implementation, this is compatible with our preferred approach of first inserting a record by the ID, and only checking the hash if the insert fails due to a duplicate entry already existing - as such I have made it the preferred option.
2. Add optional configurations to enable the Canonical JSON transformations as part of the Duplicate Check logic:
   • An additional optional config for a DateTime may be necessary to determine at what point we should apply the Canonical JSON transformations. This may be necessary to provide some backward compatibility going forward. If this config is missing, then we can immediately apply the Canonical JSON transformations (if enabled) and no date comparison is necessary. Keep in mind that existing duplicate hashed records may still be maintained past this DateTime, and this may be required to ensure that the hash is calculated correctly going forward for "old" requests.
3. The Hub Scheme could enforce that key ordering is maintained by FSPs, and no changes are required.

Request Details:

• Deadline: <When do we need a decision by?>
• Impact (Teams): <If your proposal is accepted, what team(s) will be affected by it?>
• Impact (Components): <If your proposal is accepted, which components (e.g. APIs, docs, codebases) will be affected by it?>

**Artifacts**: - [ ] Issue: https://github.com/mojaloop/project/issues/2463 - [ ] JSON Canonical RFC: https://datatracker.ietf.org/doc/html/rfc8785 **Dependencies**: - [ ] N/A ### **Accountability**: - **Owner:** @mdebarros - **Raised By:** @mdebarros ## **Decision(s)**: - **Approved By:** ### Details - [ ] Actual decision made as a result of discussion ## **Follow-up**: - [ ] Actions to implement the decisions

[lewisdaly]

• Option 1 was decided by the DA on 2021-10-06
• @pedrosousabarreto also agreed (with @lewisdaly) to follow up issued raised around the requirement for such duplicate check logic with the FSPIOP SIG