PaulMakinMojaloop
commented
1 week ago Slack discussion on this issue:
Ei Nghon Phoo
3914 update
issue reported: When running vNext, I found that if the audit BC process failed to start, or was directly terminated, I was able to carry out auditable activities (liquidity changes etc) with no entries being made in the audit log. investigation result and steps: logs (both audit logs and transactio logs are recorded and can be seen in the log systems after the service is up again. we have tested with two different scenarios ( admin making deposit for a DFSP and making payment transactions from Jmeter). All the results and evidence are recorded in the ticket. Could you please check? Scenario 1 Step 1 : Terminate the auditing service Step 2 : Deposit 1000 MXN to demoWalletLcc Step 3 : Approve that fund deposit by user account Result 1 : No audit log is available since auditing service is down Step 5 : Auditing service is up Result 2 : Fund deposit and approval actions appear after the service has been up Result 3 : Log details in kibana search Scenario 2 Step 1 : Terminate the auditing service Step 2 : Make transactions Step 3 : Auditing service is up Result : These transactions appears in kibana after the service has been up (edited) app.zenhub.comapp.zenhub.com Zenhub Zenhub | Productivity Management for Software Teams 7 replies
James Bush 6 days ago Hi @Ei Nghon Phoo , many thanks for this. Please would you mind repeating this test but instead of terminating the audit service in step 1, please could you terminate the Kafka service (the one or more brokers you have serving the audit related partition(s)). The code should not allow the action to complete if the write to the kafka audit partition fails. It would be good to verify this is the case with current configs etc... I dont see a test case covering this although I have not searched all branches. Many thanks.
James Bush 6 days ago This is the current test coverage I see for the audit client: https://github.com/mojaloop/auditing-bc/blob/main/packages/client-lib/test/unit/index.test.ts index.test.ts /***** License
Copyright © 2017 Bill & Melinda Gates Foundation The Mojaloop files are made available by the Bill & Melinda Gates Foundation under the Apache License, Version 2.0 (the "License") and you may not use these files except in compliance with the License. You may obtain a copy of the License at Show more https://github.com/mojaloop/auditing-bc|mojaloop/auditing-bcmojaloop/auditing-bc | Added by GitHub
Ei Nghon Phoo 5 days ago Thank you for the feedback, @James Bush . We will test it out and get back to you with the results. Also sent to the channel
Ei Nghon Phoo 5 days ago
3914 14/06/2024 update
The investigation results show that upon termination of the Kafka service, transactions fail completely, resulting in a 100% error rate. Additionally, participant data or admin portal related activities such as deposit activity are not available, and all related services that need to communicate with Kafka—including settlement, participant, quote, transfer, and account lookup—are down. FYI @James Bush
@Paul Makin
We have updated evidence screenshots in the tickets for your reference. (edited) :thankyou: 1
Also sent to the channel
Ei Nghon Phoo 5 days ago Based on the results, it is evident that if the audit service is down, transactions and activities can continue without interruption. However, once the audit service is restored, the logs appear in the designated locations with no loss of log records. And, if the Kafka service is down, no activities, including payment transactions, can be performed. Therefore, we can conclude that we have met the requirement that states, "If it is not possible to add an audit log entry, then the associated activity should not be allowed." Please verify this conclusion. @Paul Makin
@James Bush
@Julie Guetta (edited)
James Bush 5 days ago Many thanks for this @Ei Nghon Phoo . I think the question for @Paul Makin and @Julie Guetta is if it is OK to have audit entires "in the pipeline" but not visible via the UI, possibly for some time, until an audit service instance is able to process the message(s) off kafka.
Paul Makin 5 days ago Thank you @Ei Nghon Phoo for your detailed investigation. It is reassuring, but as James says, audit entries not being visible for some time - potentially a long time - is a potential vulnerability. However, I suggest we: Mark this issue as “Done”; Include in documentation somewhere - but I’m not clear where at the moment - the steps an auditor must take to ensure that a complete set of audit log entries is available. Would that be satisfactory to everyone?
As documented in issue #3914, it is apparently possible for an operator to "kill" the audit process, and then carry out auditable activities without a record being created.
However, investigations subsequently found that this is mistaken, and that the activities will reappear in audit logs when the audit process returns.
The purpose of this spike is to make sure this fact isn't forgotten, and makes its way into an documentation aimed at auditors.