[DMP 2024]: Survey and Polling( Django Implementation) - Backend API

[Sanchay-T]

Ticket Contents

Description

The Sp Polling Backend feature is aimed at enhancing the functionality of the Sp Polling application within the healthcare domain. This feature will streamline the authentication processes for both creators and users, introduce a forms builder, and establish critical endpoints necessary for the application's operation. The overall objective is to improve user experience and ensure secure, efficient data handling.

Goals & Mid-Point Milestone

Month 1:

• [x] Week 1: Mentor introduction, team introduction, define project goals, set up repository.
• [x] Week 2: Design project architecture (both versions), provide high-level overview.
• [x] Week 3: Implement authentication for creator scenarios.
• [x] Week 4: Implement authentication for user scenarios; begin forms builder.
• [x] Week 5: Implement important endpoints (part 1/2).
• [x] Week 6: Implement important endpoints (part 2/2).
• [x] Mid Evaluation: 29 July
• [x] Week 7-10: Work on version 2 of project (28 days, complete by end of August).
• [x] Week 11-12: Develop test cases (start on 11th September, 11 days).

Setup/Installation

No response

Expected Outcome

No response

Acceptance Criteria

No response

Implementation Details

• Django Rest Framework

Mockups/Wireframes

No response

Product Name

Sp Polling Backend

Organisation Name

a2i (aspire to innovate)

Domain

⁠Healthcare

Tech Skills Needed

Django, Python, React

Mentor(s)

@Sanchay-T

Category

API

[dvjsharma]

Weekly Learnings & Updates

Week 1

• Went through the current implementation of Survey and Polling backend API.
• Researched various system requirements, including opinion polls, election results, comments, updates, and surveys.
• Got it touch with my mentor and front-end team and decided on project goals, as well as a common aim to achieve.

Week 2

• Explored the latest stacks for potential use, with Django and FastAPI emerging as the top two contenders.
• Settled on using Django and Django REST framework along with MySQL for the backend APIs.
• Started designing a scalable architecture keeping the following things in mind:
  • APIs should be highly configurable to support multiple types of surveys and polls with different requirements, not having a specific frontend dependency.
  • The system should be able to handle multiple surveys and polls at the same time, each managed by a creator with their own set of user related rules and permissions.
  • There should be a single source of truth for all the data, and the system should be able to handle a large number of users and responses.
  • The architecture should be modular and extensible, allowing for easy integration of new features and updates.
• Below is a high-level architecture diagram of the proposed system (version 1):

• A brief understanding of the components:
  • Backend: There will be a Django based backend API layered with nginx and gunicorn for load balancing and serving the endpoints. This will be connected to a MySQL database for storing the data.
  • Frontend: The frontend will be a separate application, running in instance mode, and will be connected to the backend API for fetching and updating the data.
  • Instances: Here, instance refers to a single survey or poll, or any other entity (election, form, audit etc.) managed by a creator. I will not enforce a specific entity type in the backend, but rather provide a generic structure that can be used for multiple types of entities. This will make the system highly configurable and extensible for future use cases. The specifications of the instance are:
    • Each instance will have its own set of rules and permissions, managed by the creator.
    • The creator will be able to create, update, delete and open/close the instance, as well as view the responses and analytics.
    • The creator will also be able to set up user-related rules and permissions, such as who can participate, how many times, and what data they can see.
    • The creator will also be able to set up the survey or poll questions, options, and other details via an intuitive interface.
    • This instance will be hosted on a separate URL, and will be accessible to the users via a unique link. Backend will identify the instance based on the URL and who is accessing it (creator or user).
  • Users: Users will be able to participate in the instance based on the rules and permissions set by the creator. They will be able to submit their responses, view the results, and interact with the instance as per the creator's settings.
  • Analytics: The system will provide detailed analytics and reports for each instance, including response rates, user demographics, question-wise responses, and other relevant data. The creator will be able to view these analytics in real-time and make informed decisions based on the data.
  • Admin: There will be a separate admin panel for managing the system, including user management, instance management, analytics, and other system-related tasks. The admin will have full control over the system and will be able to monitor and manage all the activities. This is a separate component and will not be accessible to the creators or users. (currently supported in dev mode only)

Week 3

• Pushed the initial codebase for the backend API, including the basic setup, configuration, and initial endpoints. Uplink PR - #2, does the following:
  • Initial boilerplate code: Sets up the basic project structure with config as project root.
  • flake8 lint and GitHub actions: Ensures code quality and consistency both locally and via checks.
  • Dockerize the API: Facilitates deployment and testing in a consistent environment.
  • Add Django Rest Framework: Provides tools for building the API.
  • Add CORS settings: Allows cross-origin requests from the front-end.
  • MySQL configuration: Sets up the database connection for MySQL

Note: This code can be found in config project at project/config

• Developed creator authentication workflow, including their registration, activation, JWT token generation, reset password, and reset username. Uplink PR - #4, does the following:
  • Expand flake8 lint ignored files : To not give lint errors when parsing config files
  • Implement Auth Creator Workflow: base path is /api/v1
  • {{base}}/auth/users/: To create a user and send a verification email
  • {{base}}/auth/users/resend_activation/: To resend the activation email
  • {{base}}/auth/users/activation/: To activate the user
  • {{base}}/auth/jwt/create : To get the JWT token
  • {{base}}/auth/jwt/refresh : To refresh the JWT token
  • {{base}}/auth/jwt/verify : To verify the JWT token
  • {{base}}/auth/status : To check the status of the user (authenticated or not)
  • {{base}}/auth/users/ : To get the user details
  • {{base}}/auth/users/me/ : To get, modify and delete the current user details
  • {{base}}/auth/users/reset_password/ : To reset the password and send an email
  • {{base}}/auth/users/reset_password_confirm/ : To confirm the password reset
  • {{base}}/auth/users/reset_username/ : To reset the username and send an email
  • {{base}}/auth/users/reset_username_confirm/ : To confirm the username reset

Note: All of these endpoints and their corresponding functionalities can be found in core app at project/core.

Week 4

• Created the instance creation and management workflow, including the creation, update, delete, and listing of instances. Also mapped the creator to the instance and implemented the user permission workflow for all individual instances.

• Following scenarios are taken care of:

  • Creator creates a instance (name, description) -> gets back all the details including the unique hash.

  • Creator uses their JWT to access instance -> Middleware to check if the creator is accessing their instance or not.

  • Who can poll?

    • If 0x1 << 0, 'Open to All' Everyone has access via link
      • Hash sent will allow anyone to get instance data/questions
      • Backend post route is open to accept results
      • No info on who voted
    • If 0x1 << 1, 'Open within Ogranization' Only OAuth users can vote
      • Backend exposes authorize and callback routes for OAuth
      • If user is from specific org, they can vote (check from OAuth, Google groups)
      • When user signs in, they get a JWT token and their details are stored in SocialUser model. Hence we know who voted.
      • Backend post route checks instance_auth_status and accepts a JWT token for this scenario.
        • Token has user details from SocialUser model (custom made tokens) as payload.
        • To get instance data/questions, hash along with JWT token is sent. If user is valid (i.e. they complete the OAuth login, hence verifying the org), they get the data.
        • To post results, hash along with JWT token is sent. If user is valid, they can post the results and their voting status is updated. (This is a one-time thing, so that they can't vote again)
    • If 0x1 << 2, 'Open to Specific Users Only specific users can vote
      • Creator pre-defines the users who can vote
        • Creator can send bulk data to the backend in form of a CSV file or a JSON file (username and password required)
        • Creator can add/remove users from the list
        • Creator has access to all operations on the list
        • This list is unique to the instance and NOT to the creator
        • Multiple instances can have the same user list
        • Creator can download the list in form of a CSV file/JSON file.
      • Frontend exposes a login page for the user to login via username and password. If the user is in the list, they get a JWT token. (custom token, same as OAuth).
      • Backend post route checks instance_auth_status and accepts a JWT token for this scenario.
        • Token has user details from SocialUser model (custom made tokens) as payload.
        • To get instance data/questions, hash along with JWT token is sent. If user is valid (i.e. they are in the list), they get the data.
        • To post results, hash along with JWT token is sent. If user is valid, they can post the results and their voting status is updated. (This is a one-time thing, so that they can't vote again)
    • There will be an endpoint to get instance auth type(1,2,3) and status(open/closed) for the user based on the hash. It will allow anyone to get these details.

• Uplink PR - #5. Does the following:
  • Exposes more info via /admin route
  • Adds the instance creation and management workflow:
    • {{base}}/live/instance/: To get and create a new instance.
    • {{base}}/live/instance/<hash>/: To get, update and delete the instance.
    • {{base}}/live/instance/info: To get the instance auth type and status for the instance based on the hash.
  • Adds the Social User (OAuth) workflow for user authentication:
    • {{base}}/live/<str:hash>/google-oauth2/?<redirect_uri>: To authorize the user via Google OAuth for a specific instance.
    • {{base}}/live/<str:hash>/google-oauth2/?state=<state>&code=<code>: Callback route for Google OAuth to get the JWT token and register the user as a Social User for a specific instance.
    • {{base}}/live/instance/ORG/<str:hash>/: To get the list of Social ORG users for a specific instance.
    • {{base}}/live/instance/ORG/<str:hash>/download?format=<csv/json>: To download the list of users for a specific instance in JSON/CSV format.
  • Adds the User List workflow for specific users:
    • {{base}}/live/instance/CSV/<str:hash>/: To upload a CSV file with the list of users for a specific instance.
    • {{base}}/live/instance/JSON/<str:hash>/: To upload a JSON file with the list of users for a specific instance.
    • {{base}}/live/instance/CSV/<str:hash>/download: To download the list of users for a specific instance in CSV format.
    • {{base}}/live/instance/JSON/<str:hash>/download: To download the list of users for a specific instance in JSON format.
    • {{base}}/live/instance/(CSV/JSON)/<str:hash>/<str:username>: To get, update and delete a specific user from the list for a specific instance.
    • {{base}}/live/instance/<str:hash>/login: To login a user via username and password for a specific instance.
  • Creates a custom JWT backend for the Social User and User List workflows.
  • Fixes some minor bugs.

Note: All of these endpoints and their corresponding functionalities can be found in live app at project/live.

Week 5

• Provided configuration to support multiple types of databases (cloud/local) in the project. Some of them could be useful if we want live data analytics.

  • Uplink PR - #6. Does the following:
    • Adds the database configuration for MySQL, SQlite and PostgreSQL.
    • Everything is configurable via .env file.

• Developed form creation and management workflow, including the creation, update, delete, and listing of forms. Also mapped the form to the instance (and hence, the creator) and implemented the user permission workflow for all individual forms. The creators can now create forms with multiple question types and options, and have CRUD operations for the same.

  • Uplink PR - #7
    • Creates a new app data for form CRUD operations
    • Validates a form instance data and provides following operations:
      • {{base}}/data/<hash>/form/: To get and create a form mapped to a particular instance.
      • {{base}}/data/<hash>/form/<formpk>: To update and delete the form.
      • {{base}}/data/<hash>/form/<formpk>/question/: To get and create internal questions for a form.
      • {{base}}/data/<hash>/form/<formpk>/question/<quespk>: To update and delete the internal questions for a form.
      • {{base}}/data/<hash>/voter/get-data?access={{social_token}}: To allow all types of users to get form data.

Below is the form structure:

{
    "title": "General Survey ",
    "createdAt": 1720720166108,
    "fields": [
        {
            "title": "Enter your email",
            "type": "short-text",
            "required": true
        },
        {
            "title": "Enter your name",
            "required": true,
            "type": "short-text"
        },
        {
            "title": "Tell something about you",
            "required": false,
            "type": "long-text"
        },
        {
            "title": "Enter your mobile no",
            "required": false,
            "type": "number"
        },
        {
            "title": "You like cat?",
            "required": true,
            "options": [
                "yes",
                "no"
            ],
            "type": "multioption-singleanswer"
        },
        {
            "title": "Um can't decide",
            "required": true,
            "options": [
                "you decide",
                "opt a",
                "none"
            ],
            "type": "multioption-multianswer"
        },
        {
            "title": "Upload an emoji",
            "required": true,
            "type": "file",
            "accepted": [
                "jpg",
                "png"
            ]
        }
    ],
    "endMessage": "thanks for nothing",
}

The database internally breaks every form into skeleton and fields, each of them completely modular.

Note: All of these endpoints and their corresponding functionalities can be found in data app at project/data.

Week 6

• Forms added in the previous week can now accept responses from the users. The responses are stored in the database and can be fetched by the creator for analytics and reporting. The creator can also view the responses in real-time and make informed decisions based on the data. Note that multiple responses can be submitted for the same form (based on the creator's settings). Each answer istance is mapped to the question id, form id and the user id. If the auth type is anything except Open to All, the user id is stored in the response and we can track who voted.
  • Uplink PR - #8. Does the following:
    • Adds the functionality for all types of users to post responses to a form.
    • Each form can hold multiple responses mapped towards multiple answers (single response has multiple answers).
    • Admin/creator can view and delete the responses.
    • Adds the following endpoints:
      • {{base}}/data/<hash>/voter/post-data?access={{social_token}}: To allow all types of users to post form data.
      • {{base}}/data/<hash>/responses: To get all form responses.
      • {{base}}/data/<hash>/responses/<id>: To delete the specific response.

Below is the form response structure:

{
    "answers": [
        {
            "id": "16",
            "value": "hackme@gmail.com"
        },
        {
            "id": "17",
            "value": "yea its password"
        }
    ]
}

Note: For type 2 and 3 (when form is not open to all), we can also track who voted, there is a user field which is populated when any social user votes and their status turns to voted. Also, the responses only support view and delete operations to stop any kind of tampering.

Note: All of these endpoints and their corresponding functionalities can be found in data app at project/data.

Week 7

• Completed the mid-point evaluation goals and checked them through my mentor.
• Gathered feedback from the organization and cross-checked the project specific goals.
• Worked and documented extended user scenarios.
• Documented changes in response object.

Week 8

• Raised a PR - #9 which fixes the user scenario and response object bugs:
  • Adds functionality to check if all the required fields are passed or not; it not, an error is thrown.
  • Adds functionality to check and validate that one social user only votes one time.
  • Answer field now accepts JSON input, helping in single-option and multi-option.
  • Description field is added in form/skeleton to add form description.
• Added OpenAPI documentation for the APIs.

Week 9

• Worked on improving the OpenAPI documentation via configuration and adding more details. Now the documentation can serve as a guide for the users and developers to understand the API endpoints and their functionalities.

• Raised a PR - #10 which adds the OpenAPI documentation for the APIs:

  • Adds OpenAPI documentation for the APIs
  • Adds the swagger UI for the documentation
  • Anyone (user/admin/developer) can go to the <domain>:<port>/swagger to access the documentation.
  • Anyone can copy-paste the openapi.yml file into Swagger UI to access the documentation.

• Raised a PR - #11 which fixes a bug in the data app. Now the instance/form only accepts responses when all the required fields are entered.

Week 10

• Worked on deploying the system on A2I servers using apache2 and mod_wsgi

• Separately deployed the backend code on a standalone server.

• Researched, fixed and suggested solution to the following bugs:

  • There was a dotenv package parsing error.
  • The header containing auth token was not being processed.
  • The SMTP server was not sending emails.
  • New users were not being added to the database.
  • There was a python version compatibility issue.

• All the problems were resolved and the system was successfully deployed on the A2I servers.

Week 11

• Worked on developing a testing suite using pytest.
• Added a GitHub action pytest.yml to automate testing on GitHub.
• Raised a PR - #12 for the same.

Week 12

• Added unit tests for live, core and data apps.
• Verified all tests on GitHub Actions.
• Fixed a small debug value bug.
• Raised PRs - #13, #14, #15 for the respective points.
• Exposed a new endpoint /data/voter/open-forms to get links to all public surveys.

[Sanchay-T]

Weekly Goals

Week 1

• [x] Mentor and Team introduction
• [x] Define and discuss project goals
• [x] Check and run the milestones through the frontend team
• [x] Set up the repository

Week 2

• [x] Design project architecture
• [x] Provide a high-level overview
• [x] Modify the architecture to work on scale

Week 3

• [x] Set up the Django project (lint, docker, db)
• [x] Create the core & config app
• [x] Develop creator authentication workflow
• [x] Develop advanced creator auth scenarios

Week 4

• [x] Create the live app
• [x] Develop the instance creation and management workflow
• [x] Map creators to a specific instances
• [x] Allow creators to populate users for the instance
• [x] Develop 3 polling scenarios - Anyone with the link, Within the org & Creator list
• [x] Develop channels to handle social users separately

Week 5

• [x] Create the data app
• [x] Support multiple database integrations (local/cloud)
• [x] Create and map forms to an instance
• [x] Support CRUD operations on forms
• [x] Develop channels to get form data for all auth scenarios

Week 6

• [x] Complete the form handling part
• [x] Make form accept and process responses
• [x] Develop channels to post form data for all auth scenarios
• [x] Develop routes to get/edit form responses for the admin panel

Week 7

• [x] Complete mid-point evaluation
• [x] Get feedback from the organization
• [x] Work on extended user scenarios
• [x] Work on response object fields

Week 8

• [x] Research on ways to document the API
• [x] Look for tools that can manage documentation in-house
• [x] Push documentation as a separate endpoint
• [x] Fix all bugs faced by the frontend team

Week 9

• [x] Learn and understand Digital Ocean deployments
• [x] Research on which deployment platform is bet
• [x] Refactor the code to be production ready
• [x] Fix any bugs which may come up

Week 10

• [x] Deploy the backend code
• [x] Create new SMTP and other credentials for production
• [x] Fix any bugs which may come up

Week 11

• [x] Develop a unit testing suite for the codebase
• [x] Automate testing using GitHub Actions
• [x] Research on how to implement test cases

Week 12

• [x] Develop unit test cases for live, core and data app
• [x] Test the developed test cases
• [x] Solve any bugs in the process and finalize the product
• [x] Expose endpoint to get links of all public polls and surveys