[DMP 2024]: Streamline Harmony’s setup and improve documentation

[Ideonvn]

Ticket Contents

Description

Background

Harmony is an innovative open source data integration and analytics platform developed by Zenysis Technologies that stands at the forefront of combating global health challenges. It is designed to seamlessly integrate siloed data systems, thereby fostering a unified health data ecosystem. Currently, Harmony serves a critical role for 10 national governments, empowering them to make informed decisions by providing a comprehensive view of their health data landscapes. With over 1000 users, including health professionals and policymakers at all levels of the health system, Harmony is instrumental in addressing some of the world's most pressing health crises. Its capabilities have been leveraged to eliminate malaria, eradicate polio, quash cholera outbreaks, and resolve commodity stock outs among other critical health needs. The platform's versatility and impact underscore its value in the global health domain.

Need

Users encounter difficulties setting up and using Harmony due to the intricate setup process and distributed design, requiring advanced technical knowledge, which limits its accessibility and wider use.

Objective

To simplify Harmony's setup and enhance its documentation, thereby making the platform more user-friendly and accessible for new implementers.

Impact

By meeting these goals, we aim to not only lower barriers to entry for potential users but also improve the maintenance experience for existing users, thereby strengthening Harmony's role in global health initiatives.

Goals & Mid-Point Milestone

Goals

Simplify Setup Process

Activities:

• [ ] Analyze the current setup process to identify complexity and common issues.
• [ ] Collect inputs and feedback on the above from the core Zenysis development team.
• [ ] Develop a streamlined installation script or set of scripts that automate critical parts of the setup process.
• [ ] Provide implementers a sample data source to be able to quickly set up a demo pipeline.
• [ ] Test the installation scripts in various environments to ensure compatibility and ease of use.

Mid-Point Milestone:

• [ ] Have a working prototype of the installation script ready for internal testing.

Enhance Documentation

Activities:

• [ ] Audit existing README and documentation for gaps or outdated information.
• [ ] Collect inputs and feedback on the above from the core Zenysis development team.
• [ ] Create a detailed Wiki with multi language support that includes setup instructions, configuration options, use cases, and troubleshooting tips.

Mid-Point Milestone:

• [ ] Complete the initial draft of the Wiki covering setup instructions and configuration options.

Setup/Installation

Refer to https://github.com/Zenysis/Harmony?tab=readme-ov-file#project-initialization

Expected Outcome

The project to streamline Harmony’s setup and improve documentation is broken down into specific goals, each with its mid-point milestone to ensure progress within 1.5 months into the coding period.

Acceptance Criteria

Simplify Setup Process

• [ ] Installation Script Effectiveness: The new installation scripts must reduce the setup time by at least 50% compared to the manual process.
• [ ] Ease of Use: Users with minimal technical expertise should be able to execute the setup process without external assistance.
• [ ] Environment Compatibility: The installation scripts must be compatible with and executable in at least three major operating systems (e.g., Windows, macOS, Linux) on both x86 and ARM cpu architectures.
• [ ] Demo Pipeline Functionality: The provided sample data source should enable users to set up a demo pipeline successfully on the first attempt.
• [ ] Error Handling: The setup process should include clear error messages and troubleshooting steps for common issues.

Enhance Documentation

• [ ] Completeness: The new Wiki must cover all aspects of the setup process, configuration options, use cases, and troubleshooting tips, leaving no critical information gaps.
• [ ] Clarity and Accessibility: Documentation should be easily understandable by users with varying levels of technical expertise, with technical terms and processes clearly explained.
• [ ] Feedback Incorporation: The documentation must reflect inputs and feedback from the core Zenysis development team, ensuring it meets the needs of both new and existing users.
• [ ] Maintenance Guidance: Documentation includes a section on maintaining and updating the Harmony platform, with clear instructions and best practices.

General Criteria

• [ ] User Feedback: Positive feedback from a sample group of end-users and the Zenysis development team, indicating that the setup process and documentation meet their needs and expectations.
• [ ] Documentation Updates: All documentation updates must be completed and reviewed by the project team to ensure accuracy and comprehensiveness.
• [ ] Project Timeline: All mid-point milestones are achieved within 1.5 months of the coding period, and the final project deliverables are completed within the agreed-upon timeline.

Implementation Details

The Harmony project employs modern technologies such as Docker for containerization, ensuring ease of setup and environment consistency for both development and production. It integrates PostgreSQL for relational data storage and Apache Druid for handling high-performance analytics workloads, alongside MinIO for S3-compatible object storage, providing flexibility across various storage backends. Automation and simplification of development tasks are facilitated through Makefiles, streamlining the project setup and maintenance. The core solution is written in Python, using Flask as the backend web framework and React on the frontend. This technical foundation offers potential contributors a sophisticated environment for developing scalable and robust data integration solutions.

Mockups/Wireframes

No response

Product Name

Harmony

Organisation Name

Zenysis Technologies

Domain

⁠Healthcare

Tech Skills Needed

Database, Debugging, DevOps, Docker, Flask, Python

Mentor(s)

@Ideonvn @Sybrand

Category

Configuration, Documentation, Delpoyment

[Teak-Rosewood]

Hello, I am interested to contribute her as a part of DMP2024, is this issue open? and would you like me to fork the repo and start implementing my changes?

[rohitkori]

Hi @Ideonvn, I am interested in this project. I have experience working on various projects and have worked with various tech stacks like Node.js, Django, and ReactJs, as well as DevOps tools such as Docker and Kubernetes. Additionally, I am one of the coordinators of the open-source community at my college and have worked on open-source projects.

I noticed that there are several docker-compose files in the repository. Would you like to use Docker to streamline this process?

[Sybrand]

Hello, I am interested to contribute her as a part of DMP2024, is this issue open? and would you like me to fork the repo and start implementing my changes?

Hi @Teak-Rosewood .

The issue is indeed open, and you’re welcome to fork the repo and start making improvements. If you get stuck on anything, please reach out to me.

Keep in mind, the issue is pretty open ended; anything that addresses the objective (improve ease of setup) is a welcome addition.

I’d like to ensure we don’t have multiple people working on the exact same thing, so if you could please communicate which particular aspect you’d like to address 1st, that would be much appreciated. Try your best to create small, easy digestible pull requests.

[Sybrand]

Hi @Ideonvn, I am interested in this project. I have experience working on various projects and have worked with various tech stacks like Node.js, Django, and ReactJs, as well as DevOps tools such as Docker and Kubernetes. Additionally, I am one of the coordinators of the open-source community at my college and have worked on open-source projects.

I noticed that there are several docker-compose files in the repository. Would you like to use Docker to streamline this process?

Hi @rohitkori

Docker would certainly be involved; the solution currently relies on various dockerized components, builds to a docker image, and there are already some instructions around running the system localized using docker.

I would envision part of this task relating to both improving the documentation around how we use docker, and improving the use of docker itself.

[NEGRAH-S]

Hello @Sybrand @Ideonvn , i was a selected Contributor last year for C4GT and this year your project has fascinated me, I am well versed with Python, java, Docker, AWS. Am looking forward to connect and work on approach to draft a meaningful proposal for same.

[Sybrand]

Hello @Sybrand @Ideonvn , i was a selected Contributor last year for C4GT and this year your project has fascinated me, I am well versed with Python, java, Docker, AWS. Am looking forward to connect and work on approach to draft a meaningful proposal for same.

@NEGRAH-S that's great. Feel free to reach out to me, so that we can plan out some next steps.

[DhruvLamba]

Hey @Sybrand @Ideonvn , I am eager to express my interest in contributing to the Harmony project and believe I possess the skills and dedication necessary to excel in streamlining the setup process and enhancing documentation.

My background in database management, DevOps, and Docker aligns closely with the technical requirements of the project. I have extensive experience in automating deployment processes and creating scalable, containerized environments using Docker, which I believe will be invaluable in developing the streamlined installation scripts for Harmony. Additionally, my proficiency in debugging and troubleshooting will ensure that the setup process is robust, reliable, and user-friendly.

Furthermore, I have a strong track record of documentation improvement in previous projects. I understand the importance of clear, comprehensive documentation in facilitating user adoption and maintenance. Through my attention to detail and collaborative approach, I am confident in my ability to audit existing documentation, gather feedback from stakeholders, and create a detailed Wiki that addresses user needs and enhances the overall user experience.

[NEGRAH-S]

@Sybrand , how shall i reach out to you, have already dropped a connect req on LinkedIn. It would be great if you could give me your email for communication.

[Sybrand]

@Sybrand , how shall i reach out to you, have already dropped a connect req on LinkedIn. It would be great if you could give me your email for communication.

For DMP 2024 I can be reached on Discord - my user is sybrand.strauss

[Sybrand]

Hey @Sybrand @Ideonvn , I am eager to express my interest in contributing to the Harmony project and believe I possess the skills and dedication necessary to excel in streamlining the setup process and enhancing documentation.

My background in database management, DevOps, and Docker aligns closely with the technical requirements of the project. I have extensive experience in automating deployment processes and creating scalable, containerized environments using Docker, which I believe will be invaluable in developing the streamlined installation scripts for Harmony. Additionally, my proficiency in debugging and troubleshooting will ensure that the setup process is robust, reliable, and user-friendly.

Furthermore, I have a strong track record of documentation improvement in previous projects. I understand the importance of clear, comprehensive documentation in facilitating user adoption and maintenance. Through my attention to detail and collaborative approach, I am confident in my ability to audit existing documentation, gather feedback from stakeholders, and create a detailed Wiki that addresses user needs and enhances the overall user experience.

@DhruvLamba I look forward to seeing your proposal. Let me know if I you have any specific questions I can help with.

[AbhimanyuSamagra]

Do not ask process related questions about how to apply and who to contact in the above ticket. The only questions allowed are about technical aspects of the project itself. If you want help with the process, you can refer instructions listed on Unstop and any further queries can be taken up on our Discord channel titled DMP queries. Here's a Video Tutorial on how to submit a proposal for a project.

[rohitkori]

Weekly Learnings & Updates (To be filled by contributor)

Week 1

• Familiarized myself with the project's codebase and setup procedures.
• Worked on resolving an issue where bash scripts fail to execute within the container on Windows systems.

  Week 2

• Built Docker images for the development environment, troubleshooting any arising issues.
• Added a timeout in the yarn install command in the Docker_web-client file to resolve the slow network connectivity issue.
• Launched containers and debugged issues causing abrupt termination.

  Week 3

• Updated the Dockerfile to specify a compatible pip version due to Celery incompatibility with the latest release.
• Implemented dos2unix to enable bash script execution within Docker containers on Windows systems.
• Resolved issues related to running Druid in a Docker environment.

  Week 4

• Resolved issues preventing Druid from running locally.
• Ingested data into the Druid database.
• Setting up the Posgresql database for local use and troubleshooting any errors.

  Week 5

• Running Druid on a GCP instance as local development containers are unable to run due to resource limitations.
• Worked on resolving issues to successfully launch local containers.
• Trying to fix formatting problems with dummy data in Druid.

[Sybrand]

Weekly Goals (To be filled by mentor)

Week 1

• [x] Familiarise with code base + onboarding.

Week 2

• [x] Troubleshooting project setup with mentor.

Week 3

• [x] Troubleshooting project setup with mentor.

Week 4

• [x] Troubleshooting project setup with mentor.

Week 5

• [x] Troubleshooting project setup with mentor.

Week 6

• [x] Configure WSL environment for development.
• [x] "bare metal" druid instance running.
• [x] Progress on getting project up and running using "bare metal" Druid instance.

Week 7

• [x] Project up and running using "bare metal" Druid instance

. . . .

Week 12

• [x] Create PR (https://github.com/Zenysis/Harmony/pull/135) for bash script to assist with local setup

Week 13

• [x] Apply feedback to PR

Week 14

• [x] Apply feedback to PR

Week 15

• [x] Complete PR