Missed requirement/impact, now a bug, related to #154.
As per #154, the keys that are used to store and address the components of the graph related to a transaction (entity, account_holder, account, transaction) are now composed out of a number of data fields in the incoming message to ensure uniqueness across operators in a switching network. The keys are composed as follows:
Rule processors use the keys from the DataCache to retrieve transaction history from the database. Each node and edge in the pseudonyms graph database is identified directly by the keys from the DataCache:
entities node collection:
account_holder edge collection:
accounts node collection:
transactionRelationship edge collection:
pacs.008
pacs.002
For performance reasons, look-ups are done directly via the already indexed _key, _from, _to fields in the graph, whenever possible.
PROBLEM
For provenance and auditability reasons, the documents in the transactionHistory collection are written to the database "as received", without modification. The documents do not have a composite reference that can be addressed from the keys in the DataCache. Look-ups to the transactionHistory collections out of the rule processors using the dbtrId, cdtrId, dbtrAcctId and cdtrAcctId are currently failing. Once composed, the key cannot be decomposed for look-ups using its component parts.
Solution options
Add the DataCache object to the root of each document written to the transactionHistory collections, index the keys, and use this object to look up the documents.
Integrate the keys from the DataCache object into the various (Othr) data structures inside the documents as the new element [0], with a "TAZAMA_ID" scheme. I'm disqualifying this solution out of hand because the pacs.002 message doesn't have any of the ID information and this feels like too invasive an update compared to the provenance/auditability requirement.
Move the data that is required for rule processors from the transactionHistory collections into the pseudonyms database and the transactionRelationship edge collection to obviate look-ups in the transactionHistory collections. This feels cleaner, at first glance, but creates a dependency in the future where we won't be able to add any rules that rely on data that's still only in the transactionHistory collections until development work is done to first transform that data into the transactionRelationship edge collection. There is, however, a chance that we may realise a performance improvement since we'll be eliminating cross-database multi-stage reads for some of the rule processors.
Add the decomposed key components to the DataCache as well, and use these to look up documents in the transactionHistory collections.
Change the lookup approach to only use endToEndId values discovered in the pseudonyms DB to look up documents in the transactionHistory collections.
Short term (release 2.0.0)
Option 1 above would be the quickest and least invasive change.
Acceptance criteria
Out of the data preparation, after the DataCache is composed, add the DataCache object to the pain.001, pain.013, pacs.008 and pacs.002 messages before writing the messages to the transactionHistory database and the associated collections, adjacent to the root object, and along with the TxTp object that we are also inserting.
Add the DataCache keys to the indexes for each of the collections.
Update the rule queries for the rule processors that interact with the transactionHistory.transactionHistoryPacs008 collection to use the DataCache from the document for look-ups, and not the data from the Othr objects.
Missed requirement/impact, now a bug, related to #154.
As per #154, the keys that are used to store and address the components of the graph related to a transaction (entity, account_holder, account, transaction) are now composed out of a number of data fields in the incoming message to ensure uniqueness across operators in a switching network. The keys are composed as follows:
In an ISO20022 pacs.008 message, for example, the terms above are mapped from:
Debtor:
FIToFICstmrCdtTrf.CdtTrfTxInf.Dbtr.Id.PrvtId.Othr[0].Id
FIToFICstmrCdtTrf.CdtTrfTxInf.Dbtr.Id.PrvtId.Othr[0].SchmeNm.Prtry
FIToFICstmrCdtTrf.CdtTrfTxInf.DbtrAcct.Id.Othr[0].Id
FIToFICstmrCdtTrf.CdtTrfTxInf.DbtrAcct.Id.Othr[0].SchmeNm.Prtry
FIToFICstmrCdtTrf.CdtTrfTxInf.DbtrAgt.FinInstnId.ClrSysMmbId.MmbId
Creditor:
FIToFICstmrCdtTrf.CdtTrfTxInf.Cdtr.Id.PrvtId.Othr[0].Id
FIToFICstmrCdtTrf.CdtTrfTxInf.Cdtr.Id.PrvtId.Othr[0].SchmeNm.Prtry
FIToFICstmrCdtTrf.CdtTrfTxInf.CdtrAcct.Id.Othr[0].Id
FIToFICstmrCdtTrf.CdtTrfTxInf.CdtrAcct.Id.Othr[0].SchmeNm.Prtry
FIToFICstmrCdtTrf.CdtTrfTxInf.CdtrAgt.FinInstnId.ClrSysMmbId.MmbId
These keys are then packed into the
DataCache
object and transmitted to the downstream processors for evaluation.Example
DataCache
:Rule processors use the keys from the
DataCache
to retrieve transaction history from the database. Each node and edge in thepseudonyms
graph database is identified directly by the keys from theDataCache
:entities
node collection:account_holder
edge collection:accounts
node collection:transactionRelationship
edge collection:pacs.008
pacs.002
For performance reasons, look-ups are done directly via the already indexed
_key
,_from
,_to
fields in the graph, whenever possible.PROBLEM
For provenance and auditability reasons, the documents in the
transactionHistory
collection are written to the database "as received", without modification. The documents do not have a composite reference that can be addressed from the keys in theDataCache
. Look-ups to thetransactionHistory
collections out of the rule processors using thedbtrId
,cdtrId
,dbtrAcctId
andcdtrAcctId
are currently failing. Once composed, the key cannot be decomposed for look-ups using its component parts.Solution options
DataCache
object to the root of each document written to the transactionHistory collections, index the keys, and use this object to look up the documents.DataCache
object into the various (Othr
) data structures inside the documents as the new element [0], with a "TAZAMA_ID" scheme. I'm disqualifying this solution out of hand because the pacs.002 message doesn't have any of the ID information and this feels like too invasive an update compared to the provenance/auditability requirement.transactionRelationship
edge collection to obviate look-ups in the transactionHistory collections. This feels cleaner, at first glance, but creates a dependency in the future where we won't be able to add any rules that rely on data that's still only in the transactionHistory collections until development work is done to first transform that data into thetransactionRelationship
edge collection. There is, however, a chance that we may realise a performance improvement since we'll be eliminating cross-database multi-stage reads for some of the rule processors.DataCache
as well, and use these to look up documents in the transactionHistory collections.endToEndId
values discovered in the pseudonyms DB to look up documents in the transactionHistory collections.Short term (release 2.0.0)
Option 1 above would be the quickest and least invasive change.
Acceptance criteria
DataCache
is composed, add theDataCache
object to the pain.001, pain.013, pacs.008 and pacs.002 messages before writing the messages to the transactionHistory database and the associated collections, adjacent to the root object, and along with theTxTp
object that we are also inserting.DataCache
keys to the indexes for each of the collections.transactionHistory.transactionHistoryPacs008
collection to use theDataCache
from the document for look-ups, and not the data from theOthr
objects.Rules impacted
Example pacs.008
pacs.008.001.10 message with embedded `DataCache` and `TxTp` objects.
```json { "TxTp": "pacs.008.001.10", "FIToFICstmrCdtTrf": { "GrpHdr": { "MsgId": "693cd691efac4bc79e49d5ddab66e1c3", "CreDtTm": "2024-07-13T18:41:03.367Z", "NbOfTxs": 1, "SttlmInf": { "SttlmMtd": "CLRG" } }, "CdtTrfTxInf": { "PmtId": { "InstrId": "5ab4fc7355de4ef8a75b78b00a681ed2", "EndToEndId": "acc819be1a3447898bf8c00199b3029e" }, "IntrBkSttlmAmt": { "Amt": { "Amt": 301.04, "Ccy": "XTS" } }, "InstdAmt": { "Amt": { "Amt": 301.04, "Ccy": "XTS" } }, "ChrgBr": "DEBT", "ChrgsInf": { "Amt": { "Amt": 0, "Ccy": "XTS" }, "Agt": { "FinInstnId": { "ClrSysMmbId": { "MmbId": "fsp001" } } } }, "InitgPty": { "Nm": "April Blake Grant", "Id": { "PrvtId": { "DtAndPlcOfBirth": { "BirthDt": "1968-02-01", "CityOfBirth": "Unknown", "CtryOfBirth": "ZZ" }, "Othr": [ { "Id": "+27730975224", "SchmeNm": { "Prtry": "MSISDN" } } ] } }, "CtctDtls": { "MobNb": "+27-730975224" } }, "Dbtr": { "Nm": "April Blake Grant", "Id": { "PrvtId": { "DtAndPlcOfBirth": { "BirthDt": "1999-06-25", "CityOfBirth": "Unknown", "CtryOfBirth": "ZZ" }, "Othr": [ { "Id": "dbtr_9c58300b7c924ef28f685aa68efeb699", "SchmeNm": { "Prtry": "TAZAMA_EID" } } ] } }, "CtctDtls": { "MobNb": "+27-730975224" } }, "DbtrAcct": { "Id": { "Othr": [ { "Id": "dbtrAcct_c08f2262448e4798b8c1b734a1bd1974", "SchmeNm": { "Prtry": "MSISDN" } } ] }, "Nm": "April Grant" }, "DbtrAgt": { "FinInstnId": { "ClrSysMmbId": { "MmbId": "fsp001" } } }, "CdtrAgt": { "FinInstnId": { "ClrSysMmbId": { "MmbId": "fsp002" } } }, "Cdtr": { "Nm": "Felicia Easton Quill", "Id": { "PrvtId": { "DtAndPlcOfBirth": { "BirthDt": "1935-05-08", "CityOfBirth": "Unknown", "CtryOfBirth": "ZZ" }, "Othr": [ { "Id": "cdtr_c79e05f2932047a8b6ab53236afbb13e", "SchmeNm": { "Prtry": "TAZAMA_EID" } } ] } }, "CtctDtls": { "MobNb": "+27-707650428" } }, "CdtrAcct": { "Id": { "Othr": [ { "Id": "cdtrAcct_c622e7853335471bbcc8ed376dbbb073", "SchmeNm": { "Prtry": "MSISDN" } } ] }, "Nm": "Felicia Quill" }, "Purp": { "Cd": "MP2P" } }, "RgltryRptg": { "Dtls": { "Tp": "BALANCE OF PAYMENTS", "Cd": "100" } }, "RmtInf": { "Ustrd": "Generic payment description" }, "SplmtryData": { "Envlp": { "Doc": { "Xprtn": "2021-11-30T10:38:56.000Z", "InitgPty": { "Glctn": { "Lat": "-3.1609", "Long": "38.3588" } } } } } }, "DataCache": { "dbtrId": "dbtr_9c58300b7c924ef28f685aa68efeb699TAZAMA_EID", "cdtrId": "cdtr_c79e05f2932047a8b6ab53236afbb13eTAZAMA_EID", "dbtrAcctId": "dbtrAcct_c08f2262448e4798b8c1b734a1bd1974MSISDNfsp001", "cdtrAcctId": "cdtrAcct_c622e7853335471bbcc8ed376dbbb073MSISDNfsp002", "creDtTm": "2024-07-13T18:41:03.367Z", "amt": { "amt": 301.04, "ccy": "XTS" } } } ```Long term (not this ticket)
Evaluate the potential of option 3.