fatchat commented 6 months ago

Description

Dalgo is an open-source data platform for the social sector. Project Tech4Dev runs an instance of Dalgo as a commercial offering. This setup manages multiple NGOs and we need a way to keep an eye on all of them

This project will build an administrative UI for Dalgo.

Although this ticket is part of the DDP_backend repo, the selected contributor will work on a (new) frontend repo as well, and possibly on the prefect-proxy repo if required

Goals

[ ] Monitor the server by querying our existing Prometheus system
[ ] Report on jobs being run by Airbyte, how long they are taking, the machine resources they are consuming
[ ] Report on Prefect's job history, which jobs took how long, enable historical analysis of jobs by connection / by client, a view into when during the day the machine is expected to be busy
[ ] A central view into which clients use which warehouses, which data sources, how many connections, the types of sync methods selected etc

Expected Outcome

In the backend:

A new Django app under the existing Django project, connecting to the same Django database
Models and API endpoints to support the required functionality
Tests using the existing PyTest framework

A new frontend application

A new React application which talks to the Django backend
Authentication
Pages for infrastructure, Airbyte jobs, Prefect jobs ... TBD

Acceptance Criteria

[ ] The application should correctly report on the activity within the Dalgo system
[ ] Client data should not be exposed, only job & configuration data
[ ] Test coverage exceeding 80%
[ ] Developer documentation

Implementation Details

Libraries may be used but must be open-source Our backend is Django and our frontend is NextJS

Mockups / Wireframes

None but we will decide on designs before the frontend work begins

Product Name

Dalgo

Project Name

Admin Console for Dalgo

Organization Name:

Project Tech4Dev

Domain

Other

Tech Skills Needed:

Python, Django, JavaScript, NextJS

Mentor(s)

@fatchat

Complexity

High

Sub Category

API, Frontend, Backend

amansgith commented 6 months ago

Hii Sir @fatchat is it the same Admin Console that was built recently few months back but now the UI/UX needs to be developed using NextJs now??

fatchat commented 6 months ago

Hii Sir @fatchat is it the same Admin Console that was built recently few months back but now the UI/UX needs to be developed using NextJs now??

Well... yes, it is. That project got stuck because it was going to need to get all its information from Django or from Prefect, and it looked artificial being an outside project.

So this project starts by bringing it under the Django umbrella and then building from there @amansgith

AbhimanyuSamagra commented 6 months ago

Do not ask process related questions about how to apply and who to contact in the above ticket. The only questions allowed are about technical aspects of the project itself. If you want help with the process, you can refer instructions listed on Unstop and any further queries can be taken up on our Discord channel titled DMP queries. Here's a Video Tutorial on how to submit a proposal for a project.

kauxp commented 6 months ago

I am a very passionate Django Developer and I wish to work on this issue. I will work on this in the DMP timeframe and submit a proposal. If I face any technical difficulties I will contact you here!!!!

Prakhar-Shankar commented 6 months ago

Hi @fatchat ,

I've carefully reviewed the requirements outlined in the GitHub issue regarding our project goals. Based on my understanding, here's what we aim to accomplish:

Monitoring with Prometheus: Utilize our existing Prometheus system to efficiently monitor various events by crafting effective queries.
Report Generation: Generate reports for different infrastructures such as Airbyte and Perfect jobs, with the possibility of extending to other infrastructures later.
Centralized View: Develop a comprehensive view to track which clients are using which warehouse, data sources, and connections.

Additionally, at the project's conclusion, we aim for:

Backend Development: Introduce a new Django app seamlessly integrated with our existing Django database. Test the functionality rigorously using the pytest framework.
Frontend Development: Create a new frontend using React.js, serving as a dashboard for the Django backend. Ensure authentication features and incorporate pages tailored to each client's needs.

Throughout the process, we must prioritize:

Data Security: Ensure client data remains confidential and inaccessible to unauthorized users.
Test Coverage: Aim for test coverage exceeding 80% to guarantee the robustness of our system.

Before proceeding, I'd like to confirm if my interpretation aligns with the project's objectives. If so, could I begin working on a formal proposal? Looking forward for your guidance.

Best regards, Prakhar

Rishikesh63 commented 6 months ago

my tech stacks are python,django,javascript and NextJs . I wants to work on this project.

DalgoT4D / DDP_backend

Admin Console for Dalgo #561