[DMP 2024]: Mojaloop participation tools enhancement #3861

[elnyry-sam-k]

Ticket Contents

Description

There is a need to enhance, support and maintain the Mojaloop participation tools periodically. Usual maintenance and evolution to support new needs doesn’t take priority as commercial teams or teams with grants and projects focus on specific features only as required by business needs.

The Mojaloop participation tool set that includes tools such as the Mojaloop Testing Toolkit, the “Core test harness”, Miniloop, IaC need some maintenance to keep them up-to-date and enhancements to make them user friendly (to devs, qa, infra members) to use and for demonstrations for business and product users.

Goals & Mid-Point Milestone

Goals

• [x] Run participation tools successfully locally and understand operation (Mojaloop Testing Toolkit (TTK), Core test Harness (CTH), Payment manager, Miniloop (and if possible IaC later on separately)
• [ ] Update services to latest dependencies and address security issues, vulnerabilities and any fixes based on initial observation
• [x] Update Core test harness to create basic profiles to include only absolutely required steps and scripts
• [ ] Update miniloop to work with latest Mojaloop releases [by mid-point]
• [ ] Enhance demonstrations for focus use-cases and explore hosting one or more of the participation tools on cloud with separate UI if necessary
• [ ] Include finance portal on the participation tools (if within resource limits to deploy on dev machines)

Setup/Installation

Details in implementation details

Expected Outcome

• [ ] Mojaloop TTK is updated to use latest dependencies and any unused deps removed and demos enhanced where possible
• [ ] Mojaloop CTH profiles created for basic set of operations and provide support for all use cases supported by Mojaloop
• [ ] Miniloop is updated to support latest Mojaloop release (v16) and include finance portal if possible
• [ ] Create a sandbox on cloud using one of the participation tools for demonstrating Mojaloop use cases
• [ ] Create a demo setup using CTH, Payment managers separately to simulate close to real scenarios

Acceptance Criteria

No response

Implementation Details

• ML TTK: https://github.com/mojaloop/ml-testing-toolkit/tree/main
• ML CTH: https://github.com/mojaloop/ml-core-test-harness
• ML IaC: https://github.com/mojaloop/iac-modules and https://github.com/mojaloop/iacv2-docs
• ML Miniloop: https://github.com/mojaloop/mini-loop

Mockups/Wireframes

No response

Product Name

Mojaloop participation tools enhancement

Organisation Name

Mojaloop

Domain

Financial Inclusion

Tech Skills Needed

CI/CD, DevOps, Docker, JavaScript, Microservices, Node.js, React

Mentor(s)

Sam Kummary, James Bush, Paul Makin and other SMEs, community contributors as necessary

Category

Accessibility, Beginner Friendly, CI/CD, Delpoyment, Testing

[elnyry-sam-k]

Weekly Goals

Week 1

• [x] 1. Run the Core test Harness (CTH), on-premise-deploy, and Testing Toolkit locally and understand the basics
• [x] 2. In CTH, Optimize the basic P2P test by avoiding the setup of FSPs that are not required for the P2P tests; remove the - [] link 1. unnecessary profiles path: Link 1 Link 2
• [ ] 3. Review other optimizations once the above is done.
• [ ] 4. Bonus: Review adding finance portal to CTH - feasibility check

Week 2

• [x] 1. Start working on the Miniloop Project (once the target of week 1 is achieved)
• [x] 2. Run mini loop locally (locally may also mean on an AWS instance)
• [ ] 3. Update packages and ensure tests pass

Week 3

• [ ] 1. Update Mojaloop services on the Miniloop version - to use the latest releases
• [ ] 2. Review adding finance portal to mini-loop
• [ ] 3. Once mini-loop targets are achieved, reassess if we can move to work on IaC on the project (once the target of week 2 is achieved)

docker-compose.yml mojaloop/ml-core-test-harnes

Week 4

• [x] Start working on Payment Manager API
• [x] Following it, start working on Experience API and Management API once the payment manager is successfully deployed locally
• The goal is specified in Week 5.

Week 5

• [x] Start working on the Payment Manager API helm repository
• [x] Deploy it locally
• [ ] Add the test coverage to the Payment Manager which should give a 90% passed test.
• Link to Resources: R1 R2 central-ledger

Week 6

• [x] Get started with Mojaloop IaC Deployment
• [ ] Tasks to be added
• Link to resources: IaC Resource 2 Resource 3 Resource 4

Week 7

• [ ] Add Finance Portal to the Core Test Harness
• Links to resources: CTH Repo Intro to Finance Portal

[devarsh10]

Weekly Learnings and Updates

Week 1 [11/06/2024 - 14/06/2024]

• Learning 1
  1. Watched Lecture of Mojaloop on Participation Tools.
• Learning 2
  1. Understood more about the mojaloop working
• Learning 3
  1. Watched 2 videos on what the mojaloop Foundation is and how it works. Like what is DFSP, payer, payee customer, and merchant?
• Learning 4
  1. Learned about the complexity of the mojaloop architecture and how it manages where the judiciary is complex.

Week 2 [17/06/2024 - 21/06/2024]

• Learning 1
  1. Finally, I understood the objective of Mojaloop. Which is whatever DFSP you belong to, wherever payment you want to send. Mojaloop will solve the problem by enabling reliability.
• Learning 2
  1. I worked on the CMS and the issue part.
  2. Mojaloop Releases and Core Maintenance: Link
  3. Testing Toolkit & Payment Manager Updates: Link
• Learning 3
  1. Link
  2. Link
• Learning 4 Watched the following Sessions on mojaloop:
  1. Link
  2. Link
  3. TTK 1 Validating Switch Deployments Demo: Link
  4. Mojaloop Performance, Architecture PoC 22OCT20: Link

Week 3

• Learning 1

  1. Removed the files and compared them with the original repo. Didn't notice optimization. 1.1 - Raised the PR at night after deleting the JSON files which works fine.
  2. Work on the Security & vulnerability patches.

• Learning 2

  1. Mojaloop 's API: Link
  2. Learn more about FSPIOP API

• Update 1

  1. Connected with Sam Sir.
     • Learned about how the nested dependency works.
     • Learned about a few commands like npm audit and audit fix.

• Learning 3 [Thursday]

  1. Today we had a session on What is Mojaloop by Tullo Sir. He explained, how mojaloop is bringing a revolution in financial transactions, every spec of what Mojaloop is.
  2. Conclusion: We learned about all the terminologies used in Mojaloop, like DFSP, Mojaloop Collector, Mojaloop Hub/Switch, and Mojaloop API -> FSPIOP.

• Learning 4 and Update 2

  1. Done with the changes, created default-env-one and provisioning-one.json file
  2. Goal achieved, which was, nothing should get affected.
  3. That's all created new files. Instead of making changes to them.

Week 4 [01/07/2024 - 06/07/2024]

• Learning 1

  1. Today, I connected with Sam at 7 PM IST. Learned about, how to use the commands npm run test:unit for the repository ml-testing-toolkit
  2. Facing hurdles while using the command on my Windows machine.

• Learning 2 and Update 1

  1. In ml-core-test-harness repo. Till now we have increased the efficiency.
  2. Now, what to achieve is, instead of writing the whole command (docker-compose --profile all-services...up) use the command docker-compose --profile simple up which makes the deployment easy.
  3. Achieved it on [04/07/2024]

• Learning 3 and Update 2 [05/07/2024]

  1. Got the PR merger in the ml-testing-toolkit

• Learning 4 [06/07/2024]

  1. Started working on the Miniloop Tool. The objective is to make the dependencies updated.
  2. And make the mini-loop to the latest.

Week 5 [08/07/2024 - 13/07/2024]

• Learning 1

  1. Started working on Miniloop. Deployed it on an EC2 Instance(t2x.large Instance type).
  2. Learned about how the mini-loop works and the IP addresses with their Domain Name are associated in /etc/hosts file.

• Learning 2

  1. Prepared for the presentation.
  2. Watched the YouTube videos by Tom Daly. Link 1 Link 2

• Learning 3 and Update 1

  1. Implemented the learnings and attended the Online Community Meetup.
  2. Got an opportunity to present what we have done till now? and What we have planned to achieve next month.

• Learning 4

  1. Got to know about how to configure mini-loop properly and test using the existing repository Link to repo
  2. Watched this Mojaloop Session on YouTube. Session by Vijay Sir.

• Update 2 [14/07/2024]

  1. Didn't reach a solution. But found what the problem was.

Week 6 [15/07/2024 - 20/07/2024]

• Update 1 [17/07/2024]

  1. Faced the "proper" problem causing the error mentioned in the above picture. The error is related to the Readiness and Liveness Probe. Keep the mini-loop on hold for a while.

• Learning 1

  1. Learned about a few commands to exec into the container. Till now I was aware of the docker and kubectl exec only. But now, I know the command kubectl get pods <pod-name> -o=custom_column=NAME:.metadata.name,CONTAINER:.spec.container[*].name -n default which returns the container user inside the pod. So I can exec into the container and troubleshoot the error.

• Update 2 [18/07/2024]

  1. Started working on PM4ML(Payment Manager for Mojaloop Foundation). Deployed it locally.

• Learning 2

  1. Learned using JetBrains Gateway and learned about the necessary(most important) ports in the PM4ML project.
  2. This helps to configure the security group with least privileges.

Week 7 [22/07/2024 - 27/07/2024]

• Update 1

  1. Started working on on-premise-deployment. Link to GitHub Repo
  2. Deployed it locally getting stuck at, DFSP_ID and MCM_API_TOKEN. According to me.

• Update 2

  1. Get started with how things work inside the containers in on-premise-deploy
  2. Got introduced to, Manager-API and Experiece-API separate GitHub Repositories. Both of these are containers running when we run the docker-compose file in on-premise-deployment.
  3. Deployed both of them successfully in the local machine.

• Learnings

  1. Got to learn about, what is pm4ml?
  2. After it watched some youtube videos, 1, 2, [3]() on better understanding of How Mojaloop Foundation is helping FSPs to add Payment Manager to their backend. Refer to this documentation for reference.

Week 8 [29/07/2024 - 03/08/2024]

• Update 1

  Insights from my Evaluation

  1. In this week on August 1 I had an Evaluation where I mentioned what we have achieved till now in my tenure. i.e. The contribution in Core-Test-Harness and talked about the mini-loop failure and what we're planning to achieve in the next month.

  Failure I had faced in the mini-loop. I had also tried to troubleshoot.

  2. The link to the failure I faced in the Miniloop. I have linked the PDF of the errors here. Note: If you want to contribute to this Mini-loop project. Please follow the following reference.
     • Link to the YouTube Video: Link
     • Link to GitHub Repo of Miniloop: Link
     • Link to PDF, where I have mentioned the error and the troubleshooting I have done: Link

  Following are the Resources, suggested by Rahul Sir, who took my Evaluation.

  3. GitHub Repository: Link

Week 9 [05/08/2024 - 10/08/2024]

Updates

1. Had a meeting with the senior developers and we came to a few conclusions as follows,
   • Raise a PR at the Management-API GitHub repo so we can see the test coverage there.
   • Following it, take a look at IaC and Helm which will help you to better understand Management-API and Experience-API.

2. I have raised the PR. But we see no Actions were triggered. Link to the PR: Link

   Referred Central ledger and made some progress [08/08/2024]

3. So, as of now, the management-API repository is lacking the following things,
   • Test coverage command in package.json
   • .nycrc.yml file with required configuration.
   • So, I added them and made some progress. But, yet to walk a long way. The coverage summary still shows a 0% result
   • You can refer to this Video Document I created for more information related to test-coverage. I also mentioned important files required for creating test coverage.

Learnings

1. Learned about Vault and Redis.
2. Refer the central-ledger repository for better understanding of Test Coverage.
3. Learned about Test Coverage.

Week 10 [12/08/2024 - 17/08/2024]

Updates

1. Successfully able to get the test cases.[15/08/2024]
2. I have referred a few other Mojaloop repositories and concluded that we don't need the nyc(Istanbul) package to get the text summary. So, I have done this(refer to the attached image-01) in jest.config.js, and along with it, I added the text summary in the coverageReporters(image-01) which will give us the Text-Summary(image-02). Hence, we can now start focusing on, taking the threshold from 60% to 90% coverage.

Image-01: Imaeg-02:

3. Now, the focus is to make the test coverage 90%.

Learnings

1. Watched the video lecture on jest on "How to write test cases?". Link to Lecture -> JavaScript Testing with Jest – Crash Course
2. I learned more about the Jest testing framework. Link-01 Link-02

Week 11 [19/08/2024 - 25/08/2024]

Updates

1. As of now I have learned and gained some knowledge about the jest framework. I have started implementing this jest framework in two more repositories. Link-01 to PR Link-02 to PR.[21/08/2024]

Learnings

1. Alongside getting started with the IaC Project. Sam(our mentor) has organized a meeting for us with Nathan who explained to us what IaC is and how it is leveraged at Mojaloop and described all the small components referring this repository.[21/08/2024]
2. Learned about Netmaker, Lens
3. Educated me about Terraform configurations, Vault, and the ingress controller.
4. What I had done on the GitHub CodeSpaces once I forked the ML-deployment-guide is -> I had installed terraform, configured aws followed this Blog, and followed the guide.

Week 12 [26/08/2024 - 31/08/2024]

Updates

1. During this week we're getting started with the IaC projects and working on Transaction Request Service.
2. In IaC project, the objective is clear that we've to get some extensions and gaps clear, once I deploy the IaC locally.
3. Following that, in CTH our main objective is to add the service for Transaction Request Service. Here, we'll, create the service say trs-requests and all the containers will run regarding it. Done(Edit).
4. Now, after the point 3, the new objective is to add the /transactionRequest API request should reach to TTK Endpoint.

Learnings

1. During this week I have learned about, how the IaC is Implemented at Mojaloop and why it is useful at Mojaloop? and What problem IaC is solving?
2. Learned about CRD, How the Finance Portal works at mojaloop. Learned about the "Merchant requests for payment" system and few API calls and routes related to it. Like, /trasactionRequests and many others. Link to resources: BizOps Payee Initiated Transaction
3. On friday, connected with Paul Baker and Sam where, Paul taught me how can we add the Finance Portal with Core-Test-Harness.

Week 12 [02/09/2024 - 07/09/2024]

Updates

1. The goal of this week is to, get started with Finance Portal with Core Test Harness.
2. What tries I have done yet is,
   • Deployed Finance Portal UI locally, using Docker and helm charts.
   • Deployed all the components which Paul Baker taught during the call and integrated them to see how they perform.
   • After, above steps the main components I come up with is this.
   • In total there are 5 to 6 components which plays mojar role in Finance portal. Like, 1 2 3 read more....

Learnings

1. The learnings, I took during this week are, I got to learn about how the helm chart works.
2. I got used to the docker-compose file and it's terms like volumes, networks, profile and got introduced with small but necessary new terms like, user:root inside docker-compose file.