search

hackforla / HomeUniteUs

We're working with community non-profits who have a Host Home or empty bedrooms initiative to develop a workflow management tool to make the process scalable (across all providers), reduce institutional bias, and effectively capture data.
https://homeunite.us/
GNU General Public License v2.0
39 stars 21 forks source link

Convert/Port Flask code to FastAPI #789

Open lasryariel opened 2 months ago

lasryariel commented 2 months ago

Overview

The goal of this task is to migrate the existing Flask codebase to FastAPI to improve performance, flexibility, and development speed. FastAPI offers features such as asynchronous request handling, Pydantic for data validation, and better dependency injection, which will enhance our current implementation. This migration will also ensure that our project uses more modern and efficient frameworks.

Action Items

Resources/Instructions

paulespinosa commented 1 month ago

Flask auth and users endpoints

These were converted as part of #788

Coordinators endpoint

Hosts endpoint

Forms endpoints focuses on Intake Profile

The Forms API has become a submodule of Intake Profile.

The Service Providers name has changed to Housing Orgs

These were converted as part of #788. The code is located in modules/tenant_housing_orgs.

Health check endpoints

erikguntner commented 1 month ago

Thanks @paulespinosa this is super helpful.

paulespinosa commented 1 month ago

In the FastAPI migration, code has been organized according to "workflow capability" rather than "technical function." It represents the current model used to represent a Host Home Program workflow.

As we learn more, the organization of code and the choices described below will change.

Code Organization

The directories of the front-end code and the back-end code have changed as follows: Before After
The React Front-end /app /frontend
The FastAPI Back-end N/A /backend
The old Flask Back-end /api /flask-api

The new FastAPI Back-end

Under API code is now located in the /backend/app directory. Under this directory, the code is organized as follows:

The modules/ Python package (i.e. directory), contains sub-packages (sub-directories) for the "business functions." Each of the sub-directories below contain their own controllers, models, schemas, and other related code used to implement their responsibilities.

Poetry

The back-end API now uses the Python package and dependency management tool poetry https://python-poetry.org/.

The project dependencies are specified in /backend/pyproject.toml.

The poetry tool reads pyproject.toml to create virtual environments, install dependencies, and lock dependencies.

DataAccessLayer and SQLAlchemy Models

In the Flask-based API, api/openapi_server/models/database.py contained all SQLAlchemy models and a class called DataAccessLayer.

The DataAccessLayer class was not migrated. The SQLAlchemy Session is now dependency injected into path operation functions by declaring a parameter with db_session: DbSessionDep. (The name of the parameter can be any name but the type must be DBSessionDep.) For example:

# FastAPI-based API SQLAlchemy Session dependency injection
@router.get("/{housing_org_id}")
def get_housing_org(housing_org_id: int, db_session: DbSessionDep) -> schemas.HousingOrg | None:

In the FastAPI-based API - The SQLAlchemy models are moved to their related packages under modules/. For example, the SQLAlchemy model class User(Base) is now located in modules/access/models.py.

SQLAlchemy models are defined by importing Base from the core.db module. For example:

from app.core.db import Base

class User(Base):
    __tablename__ = "user"
   # ...

Migrating Models to SQLAlchemy 2.0

In the FastAPI-based API, SQLAlchemy models have been updated to using the 2.0 style declarative mappings following the steps documented in the Migrating an Existing Mapping section of the SQLAlchemy 2.0 Migration Guide.

For example, the new HousingOrgs model uses the mapped_column, Mapped type, and Annotated to create a reusable type:

intpk = Annotated[int, mapped_column(primary_key=True)]

class HousingOrg(Base):
    __tablename__ = "housing_orgs"

    housing_org_id: Mapped[intpk]
    org_name: Mapped[str] = mapped_column(String, nullable=False, unique=True)
    programs: Mapped[List["HousingProgram"]] = relationship(
        back_populates="housing_org")

Data Schemas

Data schemas represent the shape of the data that come into and go out of the API via the HTTP endpoints (a.k.a path operation functions).

In the Flask-based API, api/openapi_server/models/schema.py contained all of the data schemas. These data schemas were based off of the marshmallow library and, with the help of the marshmallow_sqlalchemy library, allowed direct conversion of SQLAlchemy models to marshmallow data schemas.

In the FastAPI-based API, the data schemas have been moved to their related packages under modules/. For example, the Flask-based API data schema class RoleSchema(SQLAlchemyAutoSchema) has been moved to the FastAPI-based API data schema modules/access/schemas.py as class RoleBase(BaseModel).

In the FastAPI-based API, marshallow is not used. pydantic is used to define the data schemas. It has the built-in ability to transform SQLAlchemy models to data schemas automatically by defining model_config = ConfigDict(from_attributes=True) in the class that defines the data schema. For example:

# FastAPI-based API data schema
class RoleBase(BaseModel):
    id: int
    type: UserRoleEnum

    model_config = ConfigDict(from_attributes=True)

CRUD and Repositories

In the Flask-based API, database access was performed directly via the SQLAlchemy Session in a controller or indirectly through a class that roughly implemented the Repository pattern.

In the FastAPI-based API, database access is performed either in a crud.py file or by using a class that implements the Repository pattern. These files are located in their related packages under modules/. For example, the modules/tenant_housing_orgs package has the file crud.py containing code used for CRUD (Create, Read, Update, Delete) operations for Housing Orgs.

For simple CRUD-like operations on a SQLAlchemy model, use a CRUD file to define the operations. For more advanced use of domain models, use of the Repository pattern is a consideration.

In either case, transactions and commits are maintained by the caller. For example, the Housing Orgs controller below maintains the database transaction. The transaction automatically commits the changes.

@router.post("/",
             status_code=status.HTTP_201_CREATED,
             response_model=schemas.HousingOrg)
def create_housing_org(
        housing_org: schemas.HousingOrg,
        request: Request,
        session: DbSessionDep) -> Any:

    with session.begin():
        db_org = crud.read_housing_org_by_name(session, housing_org.org_name)
        if db_org:
            redirect_url = request.url_for('get_housing_org',
                                           **{'housing_org_id': db_org.housing_org_id})
            return RedirectResponse(url=redirect_url,
                                    status_code=status.HTTP_303_SEE_OTHER)

        new_housing_org = models.HousingOrg(org_name=housing_org.org_name)
        crud.create_housing_org(session, new_housing_org)

    session.refresh(new_housing_org)

Controllers

In the Flask-based API, all of the controllers were located in api/openapi_server/controllers/.

In the FastAPI-base API, the controllers have been moved to their related packages under 'modules/'. For example, the Flask-based API api/openapi_server/controllers/auth_controller.py has been moved to the FastAPI-based API under modules/access/auth_controller.py.

Endpoints (a.k.a. path operation functions) are defined in the "controller" files using the FastAPI decorators. For example:

router = APIRouter()

@router.get("<endpoint path>")
@router.post("<endpoint path>")
@router.put("<endpoint path>")
@router.delete("<endpoint path>")

Dependency Injection

The FastAPI-based API uses FastAPI's dependency injection system. The dependencies are defined in modules/deps.py. FastAPI automatically injects dependencies when they are used in the parameters of path operation functions. For example:

# FastAPI-based API SQLAlchemy Session dependency injection
# DbSessionDep is defined in modules/deps.py
@router.get("/{housing_org_id}")
def get_housing_org(housing_org_id: int, db_session: DbSessionDep) -> schemas.HousingOrg | None:

Routing

The top-level router to the /api path is defined in main.py. The routes under this path are defined in modules/router.py. It defines the routes to each of the modules under modules/. FastAPI automatically finds all routers declared and used in controllers.

API Settings

In the Flask-based API, the API configuration settings were defined in api/openapi_server/configs/.

In the FastAPI-based API, the API configuration settings are located in core/config.py. It uses pydantic-settings to read environment variables or the .env file.

The Settings are available to path operation functions via dependency injection. The SettingsDep dependency is defined in modules/deps.py.

Database

In the FastAPI-based API, the SQLAlchemy database engine and session code is defined in core/db.py. Most interaction with SQLAlchemy Session or Engine will be provided via dependency injection to a controller's path operation function.

Testing

In the FastAPI-based API, tests have the sub-directories:

pytest fixtures are similar to FastAPI dependency injection system. They look the same but are written slightly different, so be aware the differences. The fixtures are defined in tests/conftest.py. A notable integration test fixture is the client fixture which is a TestClient that can be used to make calls to the HUU API.

@pytest.fixture
def client(session_factory) -> TestClient:

    def override_db_session():
        try:
            session = session_factory()
            yield session
        finally:
            session.close()

    main_api.dependency_overrides[db_session] = override_db_session
    main_api.dependency_overrides[get_cognito_client] = lambda: None

    return TestClient(main_api)

An example use of the TestClient is as follows. Note the name of the test function's parameter is the name of the fixture. pytest will automatically pass the TestClient to the test function when written this way:

def test_signin_with_fake_credentials(client):
    response = client.post(PATH + '/signin',
                           json={
                               'email': 'mdndndkde@email.com',
                               'password': '_pp#FXo;h$i~'
                           })

    body = response.json()
    assert response.status_code == 400, body
    assert body["detail"]["code"] == "UserNotFoundException", body

To mock AWS Cognito, moto is used to mock

Alembic

Alembic is a SQLAlchemy database migration tool. It is used to update database schemas in a existing environment whenever a change is deployed.

During the migration, the existing migration scripts have been deleted. This was done since all existing/older environments will be created from zero again. This includes the incubator environment.

If you have an existing Postgres container volume or SQLite database, then they will need to be deleted. The PostgreSQL container volume can be deleted using the command:

docker volume rm homeuniteus_db-data

Docker

In the migrated codebase, docker-compose.yml has been updated to contain the following containers:

The db and motoserver docker containers are now required (loosely speaking) to be running during development.

docker compose up -d --build pgadmin motoserver

The convenience container, pgadmin, transitively runs the db container.

The design of the Docker environment is pictured below.

HUU-docker-environment HUU-pgadmin4 HUU-motoserver

GitHub Actions

johnwroge commented 1 month ago

Thanks Paul. This is very helpful!

erikguntner commented 1 month ago

Thanks @paulespinosa this is awesome info!