hackforla / HomeUniteUs

We're working with community non-profits who have a Host Home or empty bedrooms initiative to develop a workflow management tool to make the process scalable (across all providers), reduce institutional bias, and effectively capture data.
https://homeunite.us/
GNU General Public License v2.0
36 stars 21 forks source link

Implement Guest Workflow Tasks Endpoints #572

Open ju1es opened 9 months ago

ju1es commented 9 months ago

Overview

The API shall manage the HUU housing Workflow. API clients (e.g. the front-end application) should be able to view Tasks (name, description, status) assigned to Guests by the Workflow. The API clients should also allow Guests to navigate to Tasks to view and work on.

Tasks are an abstract concept and their concrete implementations represent such things as Forms, Training/Lessons, Scheduling, Matching, etc. The Tasks in the Workflow are logically grouped together into Task Groups. The Task Groups represent major sub-processes of the Workflow such as "Application & Onboarding process", "Host Matching process", "Match Finalization process", etc.

For this issue, all Task Groups and Tasks up until the matching process should be returned by the API.

Task Groups and Tasks go through phases represented by the following statuses:

  1. Locked
  2. Start
  3. In progress
  4. More information is needed
  5. Complete

The API shall maintain the following invariants imposed by the Workflow:

  1. The order of the Task Groups represents the order in which the Guest must complete the Task Groups.
  2. The order of the Tasks represents the order in which the Guest must complete the Tasks within a Task Group.
  3. The Tasks in a Task Group can not be unlocked until the Task Group is unlocked and the Task before it has been completed. If it's the first Task in the Task Group, then it is automatically unlocked only if the Task Group is unlocked.
  4. A Task Group is completed only when all of its Tasks are completed. The next Task Group can then be unlocked.

Action Items

Domain concerns:

The following data must be available in the endpoint(s):

Database concerns:

Resources/Instructions

Joshua-Douglas commented 8 months ago

Overview

Create the endpoints needed by the frontend application to create and save the guest tasks. The guest onboarding process requires the guest to complete multiple tasks. Each task will have some associated data that needs to be stored. The onboarding is considered complete once all tasks are complete.

The backend should store the overall progress of the guest application, along with the detailed information associated with each task.

Requirements

In this initial PR, we will only store completed tasks. We can introduce draft task persistance in the future, this PR will be complex enough without this additional feature.

Research Questions

List the steps that we need to store in the database

List the data we need to store for each task

Great research! If possible, we should try to avoid representing these very detailed questions and answers as concrete data models. An "easy" approach would be to define a separate model for each application and add an entry for each user. Doing this, however, would be cumbersome and brittle. Adding/Removing questions would require altering our data model, and adding new applications would require creating new models. Updates to the underlying model are expensive because each time you update a database model you need to migrate previous versions. See the research sections I added below for an idea on how to achieve this.

As a starting point, we will take questions from this application https://docs.google.com/forms/d/e/1FAIpQLSc2Zm709r_7avFQYcaL9pZRwAnUknCZengn8rXP6jxx3sm9vQ/viewform?gxids=7757.

https://www.figma.com/file/BNWqZk8SHKbtN1nw8BB7VM/HUU-Everything-Figma-Nov-2022?type=design&node-id=9669-3122&mode=design&t=o5UXHReGcRoipBRO-0

Database

Do user accounts have roles associated with them?

No. This is a problem because we need to know if the user is a guest, a host, an admin, etc. to be able to dictate what actions they can take on the webapp, i.e. what endpoints they are allowed to access. This could also be a privacy concern because we need to make sure guests and hosts can only access their own data.

How do you define a new data model?

# HUU API
# /api/openapi_server/models/database.py
from sqlalchemy import create_engine
from sqlalchemy.engine import Engine
from sqlalchemy.orm import Session
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String, ForeignKey, DateTime, LargeBinary, Boolean
from os import environ as env

DATABASE_URL = env.get('DATABASE_URL')

Base = declarative_base()

class User(Base):
    __tablename__ = 'user'

    id = Column(Integer, primary_key=True, index=True)
    first_name = Column(String(80))
    last_name = Column(String(80))
    email = Column(String, unique=True, index=True)
    email_confirmed = Column(Boolean, default=False)
    password_hash = Column(String)
    date_created = Column(DateTime)
    is_admin = Column(Boolean, default=False)
    is_host = Column(Boolean, default=False)
    is_guest = Column(Boolean, default=False)

    # optional decriptive string representation
    def __repr__(self):
        return f"<User(id={self.id}, first_name={self.first_name}, last_name={self.last_name}, email={self.email}, date_created={self.date_created}, is_admin={self.is_admin}, is_host={self.is_host}, is_guest={self.is_guest})>"

What are data models used for?

# creating tables from models in db
from sqlalchemy import create_engine

engine = create_engine("sqlite://", echo=True, future=True)

Base.metadata.create_all(engine)

How do you use the new data model to actually add an entry to the database?

{
    "first_name": "Alejandro",
    "last_name": "Gomez",
    "email": "ale.gomez@hackforla.org",
    "email_confirmed": false,
    "password_hash": "lfOcifi3DoKdjfvhwlrbugvywe3495!#$%",
    "date_created": "2023-09-19 12:00:00",
    "is_admin": false,
    "is_host": false,
    "is_guest": true
}
# api/openapi_server/repository/guest_repository.py
from typing import Optional, List
from sqlalchemy.orm import Session
from openapi_server.models.database import User, DataAccessLayer
"""
inserting data into the database
"""

class GuestRepository:
    def create_guest(self, data: dict) -> Optional[User]:
        """
        create a new guest - adds data entry to the database
        """
        with DataAccessLayer.session() as session:
            new_guest = User(
                # db auto generates and auto increments an id
                first_name=data["first_name"],
                last_name=data["last_name"],
                email=data["email"],
                email_confirmed=data["email_confirmed"],
                password_hash=data["password_hash"],
                date_created=data["date_created"],
                is_admin=data["is_admin"],
                is_host=data["is_host"],
                is_guest=data["is_guest"]
            )
            session.add(new_guest)# places instance into the session
            session.commit() # writes changes to db
            session.refresh(new_guest) # erases all attributes of the instance and refreshes them with the current state of the db by emitting a SQL query. this is important for autoincrementing id
            return new_guest # returns the info from the db to the business logic 

How do you retrieve data models from the database

# api/openapi_server/repository/guest_dashboard_repository.py
from typing import Optional, List
from sqlalchemy.orm import Session
from openapi_server.models.database import User, DataAccessLayer

class GuestRepository:
    def get_guest_by_id(self, id: int) -> Optional[User]:
        """
        gets a guest by id
        """
        with DataAccessLayer.session() as session:
            return session.query(User).get(id)                      

How do you define a relationship between two tables in a database using SQLAlchemy

Good example showing a reference, but the terminology 'task' and 'subtask' is a bit misleading. A task has a specific meaning in asynchronous programming. It is typically a wrapper around a function that you want to run asynchronously.

I'm thinking instead of Task and Subtask, we can have a Dashboard, Application, & Questions. A dashboard has several ordered applications, with a progress indicator for each item.

Addressed further in a comment below. We are using task_group -> tasks

# HUU API
# /api/openapi_server/models/database.py
from sqlalchemy import create_engine
from sqlalchemy.engine import Engine
from sqlalchemy.orm import Session, relationship
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String, ForeignKey, DateTime, LargeBinary, Boolean
from os import environ as env

DATABASE_URL = env.get('DATABASE_URL')

Base = declarative_base()

class User(Base):
    __tablename__ = 'user'

    id = Column(Integer, primary_key=True, index=True)
    first_name = Column(String(80))
    last_name = Column(String(80))
    email = Column(String, unique=True, index=True)
    email_confirmed = Column(Boolean, default=False)
    password_hash = Column(String)
    date_created = Column(DateTime)
    is_admin = Column(Boolean, default=False)
    is_host = Column(Boolean, default=False)
    is_guest = Column(Boolean, default=False)

    tasks = relationship("Task", backref="user")

class Task(Base):
    __tablename__ = 'task'

    id = Column(Integer, primary_key=True, index=True)
    guest_id = Column(Integer, ForeignKey("user.id"))
    title = Column(String(80))
    status = Column(String(80))

    subtasks = relationship("Subtask", backref="task")

class Subtask(Base):
    __tablename__ = 'subtask'

    id = Column(Integer, primary_key=True, index=True)
    guest_id = Column(Integer, ForeignKey("user.id"))
    task_id = Column(Integer, ForeignKey("task.id"))
    status = Column(String(80))

How do you convert a data model into JSON?

Endpoints

Where are endpoints located within the API code?

Describe the process for adding a new endpoint

Testing

How do you run the backend test cases?

What is a pytest fixture?

Fixtures are functions that typically provide resources for the tests.

import pytest

def create_user(first_name, last_name):
    return {"first_name": first_name, "last_name": last_name}

@pytest.fixture
def new_user():
    return create_user("Alejandro", "Gomez")

@pytest.fixture
def users(new_user):
    return [create_user("Jose", "Garcia"), create_user("Juan", "Lopez"), new_user]

def test_new_user_in_users(new_user, users):
    assert new_user in users

Might not be important in this issue, but it is important to note that pytest fixtures also allow you to perform setup and teardown. You can do this using the yield keyword.

Thanks for showing this! This will be super useful

@pytest.fixture
def demo_setup_teardown():
   # Code before yield is executed before test starts
   doSetup()
   yield testData
   # Code after yield is executed after test finishes
   # Even if an exception is encountered
   doTeardown()

Show a test case that adds a value to the database and checks it

    # HomeUniteUs/api/openapi_server/test/test_service_provider_repository.py
    # Third Party
    import pytest

    # Local
    from openapi_server.models.database import DataAccessLayer
    from openapi_server.repositories.service_provider_repository import HousingProviderRepository

    @pytest.fixture
    def empty_housing_repo() -> HousingProviderRepository:
        '''
        SetUp and TearDown an empty housing repository for 
        testing purposes.
        '''
        DataAccessLayer._engine = None
        DataAccessLayer._conn_string = "sqlite:///:memory:"
        DataAccessLayer.db_init()

        yield HousingProviderRepository()

        test_engine, DataAccessLayer._engine = DataAccessLayer._engine, None 
        test_engine.dispose()

    @pytest.fixture
    def housing_repo_5_entries(empty_housing_repo: HousingProviderRepository) -> HousingProviderRepository:
        '''
        SetUp and TearDown a housing repository with five service providers. 
        The providers will have ids [1-5] and names Provider 1...Provider5
        '''
        for i in range(1, 6):
            new = empty_housing_repo.create_service_provider(f"Provider {i}")
            assert new is not None, f"Test Setup Failure! Failed to create provider {i}"
            assert new.id == i, "The test ids are expected to go from 1-5"
        yield empty_housing_repo

    # this function adds a value to the db and checks it
    def test_create_provider(empty_housing_repo: HousingProviderRepository):
        '''
        Test creating a new provider within an empty database.
        '''
        EXPECTED_NAME = "MyFancyProvider"

        newProvider = empty_housing_repo.create_service_provider(EXPECTED_NAME) # adds value to the db via a pytest.fixture of a db and HousingProviderRepository

        # checks the value returned to assert a test status
        assert newProvider is not None, "Repo create method failed"
        assert newProvider.id == 1, "Expected id 1 since this is the first created provider"
        assert newProvider.provider_name == EXPECTED_NAME, "Created provider name did not match request"

Show a test case that tests an endpoint

    # HomeUniteUs/api/openapi_server/test/test_service_provider_controller.py
    from __future__ import absolute_import

    from openapi_server.test import BaseTestCase

    class TestServiceProviderController(BaseTestCase):
        """ServiceProviderController integration test stubs"""

        def test_create_service_provider(self):
            """
            Test creating a new service provider using a 
            simulated post request. Verify that the 
            response is correct, and that the app 
            database was properly updated.
            """
            REQUESTED_PROVIDER = {
                "provider_name" : "-123ASCII&" # creates a fake test object
            }
            response = self.client.post(
                '/api/serviceProviders',
                json=REQUESTED_PROVIDER) # sends POST request with payload to API endpoint

            self.assertStatus(response, 201, # asserts the response status
                            f'Response body is: {response.json}')
            assert 'provider_name' in response.json # asserts data in response object
            assert 'id' in response.json
            assert response.json['provider_name'] == REQUESTED_PROVIDER['provider_name']

            # verifies there was a 'write' on the db, i.e. that the db was updated based on the API endpoint response
            db_entry = self.provider_repo.get_service_provider_by_id(response.json['id'])
            assert db_entry is not None, "Request succeeeded but the database was not updated!"
            assert db_entry.provider_name == REQUESTED_PROVIDER['provider_name']

Data Model Questions

Outline the relationships between Guests, Providers, Coordinators, Hosts and applications

We hope to support multiple providers. Each provider will have their own set of requirements. Ideally providers would be able to add new questions, and custom tailor their guest and host applications.

How should we design our database model?

Our design should support unique applications for each provider. If we create explicit database models for each application, then we will need to modify our database model each time an application is edited or each time a new application is added.

We can avoid restructuring our entire database model each time a routine application modification is made by abstracting the application requirements and constructing the provider applications at runtime.

To achieve this, we could store each application as table. Applications would contain a list of ordered questions. Adding a question would require adding a row to an existing application table. Adding a new application would require adding a new application table. In both cases the underlying schema would remain the same.

What does our current data model look like?

The ER diagram was generated using the ERAlchemy package. What's interesting is that most of these models are not used by our application at all.

huu_db

What should our data model look?

This design supports provider-specific applications, and would allow us to add/remove/edit applications without editing the data model.

The provider_id & role define the dashboard that the frontend will use. The dashboard defines the collection of applications that need to be completed, along with each application's progress. The application defines the set of questions that need to be asked. Each question defines the question's text and the expected response type.

Responses to the User's question are stored in the Response table. We can use this table to easily look up user responses for a given application, using the (user_id, question_id) composite key. If no response is found then we know that application has an unanswered question.

---
title: HUU Application Dashboard Data Model
---
erDiagram
    User {
        int user_id PK
        int provider_id FK
        string name
        string email UK
        enum role "Guest,Host,Coord"
    }
    Provider {
        int provider_id PK
        string name
    }
    Dashboard {
        int dashboard_id PK
        int provider_id FK
        string title
        enum role "Provider has guest & host dashboard"
    }
    Application {
        int app_id PK
        int dashboard_id FK
        int dashboard_order
        string title
        enum progress
    }
    App_Question_Junction {
        int app_id FK
        int question_id FK
        int question_order
    }
    Question {
        int question_id PK
        int text
        int response_type_id FK
    }
    ResponseType {
        int response_type_id PK
        string type_name "str, bool, int, etc"
    }
    Response {
        int user_id FK
        int question_id FK
        string response_value
        date timestamp
    }
    Provider }|--|| User : has
    Provider ||--|{ Dashboard : defines
    Dashboard ||--|{ Application :"consists of"
    Application }|--o{ Question : "utilizes"
    ResponseType ||--o{ Question : "defines" 
    User }o--o{ Response : "user answers"

Current API Questions

Show the code that is used to sign up new Guests

Do we do anything to identify Guest user accounts as 'Guest', as opposed to 'Host', 'Coordinator', 'etc'? The Applicant model provides a mechanism for associating a user with a role. It doesn't look like we use this model yet.

I think we should consider restarting our models as you suggested. Or at minimum get together and clean up (remove) unused models.

We have a model that handles the identification of user, i.e. applicant type. Here's the models used for signing up a user

# HomeUniteUs/api/openapi_server/models/database.py

class User(Base):
    __tablename__ = "user"
    id = Column(Integer, primary_key=True, index=True)
    email = Column(String, nullable=False, unique=True)

class ApplicantType(Base):
    __tablename__ = "applicant_type"

    id = Column(Integer, primary_key=True, index=True)
    applicant_type_description = Column(String, nullable=False)

class ApplicantStatus(Base):
    __tablename__ = "applicant_status"

    id = Column(Integer, primary_key=True, index=True)
    applicant_type = Column(Integer, ForeignKey('applicant_type.id'), nullable=False)
    status_description = Column(String, nullable=False)

class Applicant(Base):
    __tablename__ = "applicant"

    id = Column(Integer, primary_key=True, index=True)
    applicant_type = Column(Integer, ForeignKey('applicant_type.id'), nullable=False)
    applicant_status = Column(Integer, ForeignKey('applicant_status.id'), nullable=False)
    user = Column(Integer, ForeignKey('user.id'), nullable=False)

Here is what the code could like if we wanted to implement a new guest user, indicating their role

# HomeUniteUs/api/openapi_server/controllers/auth_controller.py

from openapi_server.models.database import DataAccessLayer, User, ApplicantType, ApplicantStatus, Applicant

def signUpGuest():  # noqa: E501
    """Signup a new Guest
    """
    if connexion.request.is_json:
        body = connexion.request.get_json()

    secret_hash = get_secret_hash(body['email'])

    # Signup user as guest
    with DataAccessLayer.session() as session:
        user = User(email=body['email'])

        applicant_type_id = session.query(ApplicantType.id).filter_by(applicant_type_description="guest").first()

        applicant_status = ApplicantStatus(
            applicant_type=applicant_type_id,
            status_description="unconfirmed_email"
        )

        session.add_all([user, applicant_status])
        # commit and refresh to db to be able to get applicant_status.id and user.id as they will be autogenerated
        session.commit()
        session.refresh(user, applicant_status)

        applicant = Applicant(
            applicant_type=applicant_type_id,
            applicant_status=applicant_status.id,
            user = user.id
        )

        session.add(applicant)

        try:
            session.commit()
        except IntegrityError:
            session.rollback()
            raise AuthError({
                "message": "A user with this email already exists."
            }, 422)

    try:
        response = userClient.sign_up(
          ClientId=COGNITO_CLIENT_ID,
          SecretHash=secret_hash,
          Username=body['email'],
          Password=body['password'],
          ClientMetadata={
              'url': ROOT_URL
          }
        )

        return response

    except botocore.exceptions.ClientError as error:
        match error.response['Error']['Code']:
            case 'UsernameExistsException': 
                msg = "A user with this email already exists."
                raise AuthError({  "message": msg }, 400)
            case 'NotAuthorizedException':
                msg = "User is already confirmed."
                raise AuthError({  "message": msg }, 400)
            case 'InvalidPasswordException':
                msg = "Password did not conform with policy"
                raise AuthError({  "message": msg }, 400)
            case 'TooManyRequestsException':
                msg = "Too many requests made. Please wait before trying again."
                raise AuthError({  "message": msg }, 400)
            case _:
                msg = "An unexpected error occurred."
                raise AuthError({  "message": msg }, 400)
    except botocore.excepts.ParameterValidationError as error:
        msg = f"The parameters you provided are incorrect: {error}"
        raise AuthError({"message": msg}, 500)

How should the frontend and backend interact?

The plan is to implement endpoints that the frontend can use to query a user-specific dashboard populated with applications & their progress. The dashboard will contain app_ids that can be used to load and save the application in the backend. Each application will contain an ordered list of questions and responses

---
title: Application Logic Flow 
---
flowchart TB;

subgraph frontend;
    f1["Client\nSign in"]
    f2["Load page\nusing role"]
    f3["Show Dashboard\n & Wait"]
    f4["Dashboard\nApp OnClick"]
    f5["Display\nApp"]
    f6["App\nOnSave"]
    f1 ~~~ f2 ~~~ f3 ~~~ f4 ~~~ f5 ~~~ f6
end

subgraph backend;
    b1["sign-in()"]
    b2["app_dashboard()"]
    b3["get_app"]
    b4["update_app()"]
    b1 ~~~ b2 ~~~ b3 ~~~ b4
end

f1 --"POST\n{\nusername,\n password\n}"--> b1 
b1 --"{\n&nbsp;&nbsp;role\n&nbsp;&nbsp;jwt\n}"--> f2
f2 --"GET\n/api/dashboard"--> b2
b2 --"{\napp1 {\nprogress,\ntext,\napp_id\m},\napp...\n}"--> f3
f4 --"GET\n/api/application/app_id"--> b3
b3 --"{\nprogress\nquestion1{\n&nbsp;&nbsp;progress,\n&nbsp;&nbsp;text,\n&nbsp;&nbsp;response_type,\n&nbsp;&nbsp;response\n},\nquestion2...\n}"--> f5
f6 --"PUT\n{\nprogress\nquestion1{\n&nbsp;&nbsp;progress,\n&nbsp;&nbsp;text,\n&nbsp;&nbsp;response_type,\n&nbsp;&nbsp;response\n},\nquestion2...\n}"--> b4
linkStyle 8,9,10,11,12,13,14 text-align:left

General Solution Plan

Implement a flexible application model backend system, that will allow the front-end app to query Guest and Host applications using a user_id. See the model erdiagram and application flow diagram above.

Each user will be associated with a single provider. Each provider will have a unique guest and host application. The front-end application will receive all the information it needs to dynamically generate guest and host applications.

Our backend has a 'demo' database model, however many of the models are currently unused by the frontend application. We need to decide if we want to start fresh, or if want to modify the existing model to meet our current needs. Modifying an unused model can be very challenging since we do not know if the current unused model works. I propose removing the unused models and enforcing a requirement that all new models need to be used by either the frontend app or our test project.

Model

We will implement the model by starting at the user and working our way towards responses. We will create test cases to exercise the new models at each step.

  1. Introduce a Role enumerated type with Guest, Host, Coordinator, and Admin roles
  2. Update the User model to include a role and provider_id
  3. Update the signup methods to specify a role when the user signs up
  4. Add a ResponseType model that defines the supported response types. The frontend will rely on this to interpret and edit the question responses
  5. Add a Question model to store the bank of questions. The question bank will be shared across applications.
  6. Add a Dashboard model to store a provider's role-based dashboards.
  7. Add a Application model to store the dashboard's ordered list of applications
  8. Add a AppQuestionJunction table to store the many-to-many relationship between each application and the ordered list of questions

Database Migration

With these changes we will need to store the dashboard and application structure within the database, since this structure can be defined and edited by each provider.

  1. Write a script that can create a development database. This development database should contain:
    1. A default provider
    2. A default Guest Dashboard
    3. A default Host Dashboard
    4. Applications & Questions used by the dashboards
  2. Write a migration script that can update our existing database to use the new model.
  3. Remove all unused models to simplify future development

Endpoint

  1. Add a app_dashboard() endpoint
    • Retrieve the dashboard using a user_id and provider_id
    • Return a dashboard as json, containing an ordered list of applications. Each application will include a progress field and an app_id that can be used to query the application
  2. Add a get_app() endpoint
    • Retrieve the application using the app_id
    • Populate the response for each question by querying the Response table using the user_id and question_id. If no response is found then the user has not answered that question. If a response is found, then return it within the application.
  3. Add a update_app() endpoint
    • PUT the same basic json received from the get_app() endpoint, except the response_value fields will contain user responses

Frontend

This will be apart of a separate issue.

Implementation Questions

Joshua-Douglas commented 8 months ago

Hey @agosmou,

An initial project plan is ready for this issue. Please submit your answers and reassign me once it is ready for review.

My comments are included as

quoted text

Please don't remove those section. I'll remove them as I review your responses.

agosmou commented 8 months ago

Working through this on markdown doc in VSCode. Ill paste it in here when I finish over the next day or so.

erikguntner commented 8 months ago

@agosmou @Joshua-Douglas I simplified the dashboard data a bit and wanted to leave an example here for reference incase it's helpful.

Sample response for tasks and sub-tasks:

  {
    id: 1,
    title: 'Application and Onboarding',
    status: 'in-progress',
    subTasks: [
      {
        id: 1,
        title: 'Application',
        status: 'complete',
        description:
          'Start your guest application to move on to the next step.',
        buttonText: 'Start application',
        url: '/guest-application',
      },
      {
        id: 2,
        title: 'Coordinator Interview',
        status: 'in-progress',
        description: 'Meet with your Coordinator to share more about yourself.',
        buttonText: 'Schedule interview',
        url: '/schedule',
      },
      {
        id: 3,
        title: 'Training Session',
        status: 'locked',
        description:
          'Complete a training session to prepare you for the host home experience.',
        buttonText: 'Schedule training',
        url: '/schedule',
      },
    ],
  },

Potential entity relation diagram:

erDiagram
    USER ||--|{ TASK : has
    TASK {
        int id PK
        int userId FK
        string title
        int statusId FK 
    }
    TASK ||--|{ SUB_TASK: contains
    TASK ||--|| STATUS: has
    SUB_TASK ||--|| STATUS: has
    SUB_TASK{
        int id PK
        int taskId FK
        string title
        string description
        int statusId FK
        string buttonText 
        url string
    }
    STATUS {
       int id PK
       string status "locked, inProgress, complete" 
    }
agosmou commented 8 months ago

@erikguntner

Awesome info. Thanks Eric! I'm wrapping up the design doc. I'll post this afternoon.

erikguntner commented 8 months ago

@agosmou this looks awesome! Just FYI Cognito tracks whether the user is confirmed in case that changes whether we want to keep that info in the database as well or not

Joshua-Douglas commented 8 months ago

Hey @agosmou, Thanks for the designs! I'll make sure to review them by Wednesday night!

agosmou commented 8 months ago

Figma Designs for backend-design consideration

@Joshua-Douglas - I can review these tomorrow night to see if there are any effects on the above

https://www.figma.com/file/BNWqZk8SHKbtN1nw8BB7VM/HUU-Everything-Figma-Nov-2022?type=design&node-id=9669-3122&mode=design&t=oPs2UK9Ee4GFVwK4-0

Joshua-Douglas commented 8 months ago

Hey @erikguntner,

Thanks a lot for the Task/SubTask idea. After reading through @agosmou's research I think a generic approach like this is going to be easier to implement than the more 'naive' approach of defining each application as a separate model.

It would be great if you could look through the proposed data model & application flowchart I posted above. If the frontend is already relying on the backend to query the dashboard and applications then that gives us the flexibility to define provider-specific and role-specific dashboards. This would mean that each individual provider could define a custom Guest and Host dashboards, complete with custom applications.

Joshua-Douglas commented 8 months ago

Hey @agosmou,

The design document is ready for review! Your research looked great, and it inspired me to think up a generic approach that could accommodate the Task/Subtask approach outlined by @erikguntner.

I left

comments

on some of your responses, and added several new sections (everything after the Data Model Questions section). I left one question un-answered, so I could really use your help getting to the bottom of it.

Can you answer that question, and review the design? I'd like to get your feedback. I'm sure something is missing or could be improved. If you agree with the general approach then I'll create a set of implementation questions that we can use to outline the trickiest parts of the implementation before opening a branch.

paulespinosa commented 8 months ago

@agosmou The Guest Dashboard story #500 uses Dashboard -> Steps -> Tasks language to describe the Guest Dashboard (see the "Key Decisions" section). In the diagram provided by @erikguntner's, Task should be "Step" and Sub_Task should be "Task" to align with the story's description of the Guest Dashboard feature. If anyone has found it makes more sense to name these entities differently, loop @sanya301 in to iterate on the language we'll use to describe and implement the feature.

The Dashboard Application Model diagram provided by @Joshua-Douglas is a little bit out of scope for this feature but still eventually necessary. It would need to be modified to associate the Application model with a Task: A Dashboard contains Steps, Steps are a group of Tasks, Task has-a/is-a:

Is the above a correct representation of the Guest Dashboard model?

Joshua-Douglas commented 8 months ago

Hey @paulespinosa, How is the guest dashboard data model diagram out of scope for the guest dashboard endpoint issue? Do you think we should split the two issues and backlog this one, or do you think that changes to the data model are not strictly necessary?

paulespinosa commented 7 months ago

@Joshua-Douglas specifically, the "HUU Application Dashboard Data Model" diagram. It contains models (Application, Question, ResponseType, Response, App_Question_Junction), that are circled in the image below, that are more representative of an application form and are great candidate models for the issues: Guest Application Form #533 and Guest application implementation #610 and beyond.

pic-selected-231006-2156-34

Caveat: I've been using the terminology "Steps" and "Tasks" as defined in the main issue Guest Dashboard #500. However, it appears terminology has evolved from there, so I've pinged Sanya https://github.com/hackforla/HomeUniteUs/issues/500#issuecomment-1751054169 to motivate alignment on how the team describes the Dashboard features.

The Guest Dashboard #500 issue, the parent issue for this issue, Implement Guest Dashboard Endpoints #572, specifies displaying "high-level" information such as status, description, and progress about Steps. It's not asking to display the details of an application (form). When #500 talks about being able to "click into any additional page(s)", it means as a generic action where the details of those landing pages are unspecified and left for implementation in separate issues.

So the question becomes: how do you get the status and progress of a Step? Perhaps, by asking each of its Tasks for their status and progress. How do you get the status and progress of a Task representing, say, an Application Form? Consider, as an implementation option, defining Task as an interface or abstract base class. Have concrete Dashboard context representations of Tasks, such as an Application Form, contain the logic to calculate or get the status and progress. For this issue, the concrete implementations (or simply the Task for now) can return canned responses until specific issues are defined to implement their details (e.g. regarding Application Form, it can be an issue on its own or an action item for implementing the backend for Guest Application Form #533).

Unfortunately, the Action Items in this issue appears to be misaligned with its parent. pic-selected-231007-0015-32 Based on the context of #500, the action items circled above should probably read:

Regarding #500's Consideration: "In the future, the plan is for the guest onboarding experience for multiple organizations (SPY and beyond), where different organizations might want to modify the steps needed for a guest.". This word "modify" is under-specified and needs to be discussed with Product Management https://github.com/hackforla/HomeUniteUs/issues/500#issuecomment-1751054169.

agosmou commented 7 months ago

According to this comment, we are doing the following naming convention:

"Task Groups has Tasks"


I think we can redefine the scope above to limit this issue to guest dashboard task groups and tasks, so we can remove the extra action items. We can use all the models above on issue #500 and develop them more further on their respective issues.

agosmou commented 7 months ago

@Joshua-Douglas Take a look at the edits above on the design doc. Let me know if you want to meet to discuss our models further - the ERDiagram you posted puts into perspective the reorganizing that has to be done

agosmou commented 6 months ago

cc: @tylerthome

Hi @paulespinosa - Are you able to look over my progress setting up models? I want to make sure Im understanding this and going in the right direction.

I tried my hand at what the models would look like so I could then work on the tests, domain objects, and domain logic.

As far as the statemachine (state chart pattern), we will have models.py and statechart.py that has the logic for this. At this time, I am planning on handrolling unless we agree to using a library.

models to be reviewed

##########################
# api/models/database.py #
##########################
import enum # adds python enum import
from sqlalchemy import create_engine
from sqlalchemy.engine import Engine
from sqlalchemy.orm import Session, declarative_base
from sqlalchemy import Column, Integer, String, ForeignKey, DateTime, LargeBinary, Enum # adds enum
from sqlalchemy.orm import relationship # adds imports

Base = declarative_base()

# proposed models...

class TaskStatus(enum.Enum):
    LOCKED = 'Locked'
    START = 'Start'
    IN_PROGRESS = 'In Progress'
    MORE_INFORMATION_NEEDED = 'More Information Needed'
    COMPLETE = 'Complete'

class TaskGroupStatus(enum.Enum):
    LOCKED = 'Locked'
    START = 'Start'
    IN_PROGRESS = 'In Progress'
    MORE_INFORMATION_NEEDED = 'More Information Needed'
    COMPLETE = 'Complete'

class TaskGroupSubProcess(enum.Enum):
    LOCKED = 'Locked'
    APPLICATION_AND_ONBOARDING_PROCESS = 'Application & Onboarding Process'
    HOST_MATCHING_PROCESS = 'Host Matching Process'
    MATCH_FINALIZATION_PROCESS = 'Match Finalization Process'

class Guest(Base):
    __tablename__ = 'guest'
    id = Column(Integer, primary_key=True, index=True)
    email = Column(String, ForeignKey('user.email'), nullable=False)
    tasks = relationship("Task", back_populates="guest", cascade="all, delete")
    task_groups = relationship("TaskGroup", back_populates="guest", cascade="all, delete")
    guest_workflows = relationship("GuestWorkflow", back_populates="guest", cascade="all, delete")

class Task(Base):
    __tablename__ = 'task'
    id = Column(Integer, primary_key=True, index=True)
    title = Column(String, nullable=False)
    description = Column(String, nullable=False)
    task_group_id = Column(Integer, ForeignKey('task_group.id'))
    status = Column(Enum(TaskStatus), default=TaskStatus.LOCKED, nullable=False) # from finite state machine (statechart pattern)
    guest_id = Column(Integer, ForeignKey('guest.id', ondelete='CASCADE'))

    # methods for state machine (statechart pattern)

class TaskGroup(Base):
    __tablename__ = 'task_group'
    id = Column(Integer, primary_key=True, index=True)
    name = Column(Enum(TaskGroupSubProcess), default=TaskGroupSubProcess.LOCKED, nullable=False) # from finite state machine (statechart pattern)
    guest_workflow_id = Column(Integer, ForeignKey('guest_workflow.id'))
    status = Column(Enum(TaskGroupStatus), default=TaskGroupStatus.LOCKED,nullable=False) # from finite state machine (statechart pattern)
    guest_id = Column(Integer, ForeignKey('guest.id', ondelete='CASCADE'))
    guest = relationship("Guest", back_populates="task_groups")
    tasks = relationship("Task", back_populates="task_groups")

    # methods for state machine (statechart pattern)

class GuestWorkflow(Base):
    __tablename__ = 'guest_workflow'
    id = Column(Integer, primary_key=True, index=True)
    guest_id = Column(Integer, ForeignKey('guest.id', ondelete='CASCADE'))
    task_groups = relationship('TaskGroup', back_populates='guest_workflow')
    guest = relationship("Guest", back_populates="guest_workflows")

# existing code..

class User(Base):
    __tablename__ = "user"
    id = Column(Integer, primary_key=True, index=True)
    email = Column(String, nullable=False, unique=True)

class ApplicantType(Base):
    __tablename__ = "applicant_type"

    id = Column(Integer, primary_key=True, index=True)
    applicant_type_description = Column(String, nullable=False)

class Applicant(Base): 
    __tablename__ = "applicant"

    id = Column(Integer, primary_key=True, index=True)
    applicant_type = Column(Integer, ForeignKey('applicant_type.id'), nullable=False)
    applicant_status = Column(Integer, ForeignKey('applicant_status.id'), nullable=False)
    user = Column(Integer, ForeignKey('user.id'), nullable=False)

# rest of models here...

class DataAccessLayer:
    _engine: Engine = None

    @classmethod
    def db_init(cls, conn_string):
        cls._engine = create_engine(conn_string, echo=True, future=True)
        Base.metadata.create_all(bind=cls._engine)

    @classmethod 
    def session(cls) -> Session:
        return Session(cls._engine)
paulespinosa commented 6 months ago

Hi @agosmou PM has been working on statuses in a Google Doc at https://docs.google.com/document/d/1mksBNqE9hc-bAW49mdHQDImkk8f92YewtwENNpqvFHc/edit. We (devs) will need to work together with PM on defining the statuses. As of this writing, it looks like the statuses in the Google Doc are "user statuses". It's not yet clear how the relation between "User status" and "Task status" will work.

I think the Guest should not contain references to its Tasks or TaskGroups; they should be obtained via the GuestWorkflow or similar. With respect to Guest holding a reference to the GuestWorkflow, that can be done or the GuestWorkflow can be obtained via another mechanism; we won't really know which suits us until we start working with the code and gain more clarity on the domain.

TaskStatus, TaskGroupStatus, TaskGroupSubProcess are ok to hard code for starters but will need to be stored in the DB.

Try hand rolling a basic state machine to get a feel for it and gain better clarity about our needs. Thank you.

agosmou commented 5 months ago

cc: @tylerthome

Thanks for this info, @paulespinosa ! Given the 'Host' items also take status, it'd be good to keep this in mind to make the code reusable for other endpoints.

Below Iadjusted the models and made a quick run at a state machine

Adjusted Models

#models/database.py

# proposed models...

class TaskStatus(enum.Enum):
    LOCKED = "Locked"
    START = "Start"
    IN_PROGRESS = "In Progress"
    MORE_INFORMATION_NEEDED = "More Information Needed"
    COMPLETE = "Complete"

class TaskGroupStatus(enum.Enum):
    LOCKED = "Locked"
    START = "Start"
    IN_PROGRESS = "In Progress"
    MORE_INFORMATION_NEEDED = "More Information Needed"
    COMPLETE = "Complete"

class TaskGroupSubProcess(enum.Enum):
    LOCKED = "Locked"
    APPLICATION_AND_ONBOARDING_PROCESS = "Application & Onboarding Process"
    HOST_MATCHING_PROCESS = "Host Matching Process"
    MATCH_FINALIZATION_PROCESS = "Match Finalization Process"

class Guest(Base):
    __tablename__ = "guest"
    id = Column(Integer, primary_key=True, index=True)
    email = Column(String, ForeignKey("user.email"), nullable=False)
    guest_workflows = relationship(
        "GuestWorkflow", back_populates="guest", cascade="all, delete"
    )

class Task(Base):
    __tablename__ = "task"
    id = Column(Integer, primary_key=True, index=True)
    title = Column(String, nullable=False)
    description = Column(String, nullable=False)
    task_group_id = Column(Integer, ForeignKey("task_group.id"))
    status = Column(
        Enum(TaskStatus), default=TaskStatus.LOCKED, nullable=False
    )  # from finite state machine (statechart pattern)

    # methods for state machine (statechart pattern)

State Machine (State Chart Pattern)

pseudo-implementation of hand rolled state machine using state chart pattern

class TaskGroupStateMachine: def init(self, task_group_id, session: Session): self.task_group = session.query(TaskGroup).get(task_group_id)

    self.session = session
    self.state = self.task_group.status

def transition(self, trigger):
    match trigger:
        case "start":
            self.state = TaskGroupStatus.START
        case "progress":
            self.state = TaskGroupStatus.IN_PROGRESS
        case "need_info":
            self.state = TaskGroupStatus.MORE_INFORMATION_NEEDED
        case "complete":
            self.state = TaskGroupStatus.COMPLETE

    print(f"Transitioning task group to {self.state}")
    self.task_group.status = self.state
    self.session.commit()


## Design
[Figma file](https://www.figma.com/file/BNWqZk8SHKbtN1nw8BB7VM/HUU-Everything-Figma-Nov-2022?type=design&node-id=1483-15519&mode=design&t=zmiPWI8j6AnuwXJW-0)

## UI Design
`~Ready for hand off - Section 2 guest dashboard`
![image](https://github.com/hackforla/HomeUniteUs/assets/73868258/6f12e775-122d-4c5b-b137-3ab6badd8237)

## Entities
`~Brainstorm - Sandbox`
![image](https://github.com/hackforla/HomeUniteUs/assets/73868258/bffbd24d-0d2b-44e3-9eba-fa42a3c9da0a)
tylerthome commented 4 days ago

This should be ice-boxed at least until existing MVP scope is nominally complete -- this is a requirement for compliance and traceability, but not strictly required for stakeholder demo