[DISCUSSION] Backend decisions/architecture guidelines

[leothomas]

Background

Since this is a bit of a green-field project I wanted to jot down some notes about how to structure the project.

Some guiding principles:

• Should re-use existing code libraries as much as possible (pip install > forking > re-writing from scratch)
• Should be easy to deploy as a single unit
• Should be easy to re-use constituent parts in another project

Three constituent parts are:

• A tiling api (TiTiler)
• A STAC api (stac-fastapi)
• An ingestion pipeline (Custom implementation)

Everything after this line is up for discussion and debate.

Tiling API:

Option 1: TiTiler as an external dependency

TiTiler is available as a pip-installable dependency. To add the TiTiler routes to an existing FastAPI application:

from titiler.core.factory import TilerFactory

# Create a FastAPI application
app = FastAPI(
    description="A lightweight Cloud Optimized GeoTIFF tile server",
)

# Create a set of COG endpoints
cog = TilerFactory()

# Register the COG endpoints to the application
app.include_router(cog.router, tags=["Cloud Optimized GeoTIFF"])

This will include the following routes: https://developmentseed.org/titiler/endpoints/cog/

This makes the Tiling functionality very easy and straightforward to add as an external dependency to the project - which reduces the amount of code we have write and maintain and ensures that we are able to easily stay up to date with developments to the tiler.

Option 2: Fork TiTiler

I'm not sure why we would want to do this, but I'm including as an option worthy of discussion

STAC API:

Some unknowns about stac-fastapi:

• Is there an out-of-the-box integration with TiTiler somehow?
  • If not, we should fork the stac-fastapi repo, add TiTiler as a pip dependency and add the /cog routes using the method defined above
• How do we deploy stac-fastapi using CDK? I see 2 principle options (which will be discussed further down)
  1. (I will describe this option in further detail than the other, because I'm more familiar it, but by no means does this mean I think its the better of the two options). Add a lambda handler file, which imports the FastAPI app and wraps it with Mangum. Extend the Dockerfile included in the stac-fastapi repo to copy the handler file to the Docker execution context. Deploy the stac-fastapi app as a lambda function with an API Gateway integration, and an RDS instance to host the postgres DB
  2. Deploy the Docker image included in stac-fastapi as an ECS task with an Elastic Load Balancer to provide a public interface to the ECS service.
     • Do we add a Fargate layer for orchestration? (If so, how do Fargate and ECS interact)
     • ELB vs ALB? Is there a difference? Is one better than the other?
     • The RDS instance will likely be deployed within a VPC - how can we ensure the ECS service is able to access the RDS instance inside the VPC?
     • What kind of features does ELB/ALB have that we can take advantage of? Logging? Rate limiting? Role-based authentication?
     • For reference the csdap-orders STAC API is deployed as an ECS service with an ELB

pg_stac:

stac-fastapi is built on top of pg_stac. pg_stac provides the postgres schemas and functions necessary for implementation of the STAC API.

The pg_stac schemas/functions can be applied to the database using 2 options:

• sqitch
• sqlalchemy alembic

I believe the preferred method to apply the schemas/functions is using alembic, which can be run within a python script, using the sqlalchemy psycopg2 database connector.

The csdap-orders STAC API CDK deployment code has an example of defining a custom resource within the CDK stack, where the custom resource is the actual execution of a lambda function, in this case, the lambda function runs the database setup (or "bootstrapping") using alembic. I'm a little bit fuzzy on the details, but I believe this is the overall idea.

Unknowns:

• pg_stac comes with a python utility (pypgstac) for loading data and migrating the database from one version of postgres to the next. Does this utility also contain functionality to "bootstrap" the database?
• Is there an example/implementation somewhere of "bootstrapping" a pg_stac database?

TiTiler-pgstac:

The titiler-pgstac is an extension for TiTiler that connects directly to a pg_stac database to generate mosaics dynamically from STAC queries. This may be an interesting options to explore.

Unknowns:

• Does this extension provide a STAC API endpoint on the tiler? Or just the ability to generate mosaic's from STAC queries?

Due to STAC's widespread adoption and strong specification, we want the tiler to work with any STAC API/database. This means that we would like the tiler to be able to generate mosaics from a STAC API endpoint, rather than directly accessing the database. If the TiTiler-pgstac extension provides a STAC API endpoint, it would be easy to configure the tiler to access the STAC records through any API endpoint, enabling this backend for both scientists that want to build and manage their own STAC API or scientists that already have a STAC API up and running.

If the Titiler-pgstac does not provide a STAC API endpoint then we should stay away from this implementation in order to enable the tiler to be compatible with any STAC API.

API Gateway + Lambda vs ELB + ECS:

Paraphrasing Drew: we used to prefer Lambda for the near infinite scalability, and for the lack of costs when there's no traffic to the application. Some drawbacks to Lambda include limited execution time and memory, inability to make use of multi-threading or to re-use database connections to speed up operations (since each execution may or may not take place in a new container instance). An example of this: GDAL natively implements some cacheing mechanisms (1, 2) in order to serve adjacent tiles simultaneously (or at least faster). This cacheing functionality is lost when running the tileing logic in Lambda, since each request might be served by a different container, which means that each tile read must wait for the file lock to be released before locking the file itself, transforming the simultaneous reads that are made possible by GDAL into a single threaded operation.

One way to get around these issues is to implement intelligent cacheing in the lambda function. (Note: It's important not to directly cache the result of an API call with the associated URL parameters, since some of the parameters only affect the display options of the tile, but rather cache the actual data that gets pulled from S3 - this way a single cached tile can serve multiple requests with different visualization parameters)

If partners accept the (relatively small) cost of ECS services at rest, since ECS has to have at least one EC2 instance running at any given time, and if the scaling rules are well defined, ECS can be favorable for the reasons listed above: better use of multi-threading, re-using database connections, ability to handle larger data tiles/longer runtime operations, etc without timing out.

For all the reasons above, ECS seems to be a better option for the tiling API. If the STAC API requests are small, self contained, and not memory intensive, it may make sense to implement the STAC as a Lambda + API Gateway stack. In this case, it might make sense to have 2 separate FastAPI app's (the tiler deployed as ECS + ELB and the STAC API deloyed at API Gateway + Lambda). Although this has the obvious disadvantage of requiring 2x as much infrastructure to manage.

A distinct advantage of API Gateway is the out-of-the-box integration with cognito user pools for user authentication/authorization (which would ensure that unauthorized users cannot write or edit records to the STAC database). I believe it might also be possible to integrated API Gateway with an ELB.

Ingestion pipeline:

The other large part of the backend is an ingestion pipeline that allows scientists to upload their data. At a minimum, this pipeline should do the the following for any uploaded datafile:

• Validate that the file contains is a properly formatted COG (rio coego validate), and has a nodata value set
• Create a STAC record in the database with the metadata + s3 location of the file

With successive iterations, this pipeline can add additional processing options:

• Convert from GeoTIFF to COG
• Convert from NetCDF to COG
• Unzip archives of COG/GeoTIFF's before converting (if necessary) and ingesting (In all three of the above cases, the ingestion mechanism should also copy the original, unprocessed data to an archive for safekeeping/re-ingestion)
• Restrict access for each scientist/science group to a single "delivery" subfolder
• Notify uploading users if their data has been rejected
• Enable "B.Y.O.D" (Bring Your Own Dockerfile) ingestion pipeline where scientists setting up the pipeline can configure any number of different ingestion triggers, where each trigger would execute a customizable processing by packaging the user's Dockerfile as the processing lambda function.

Architecture:

The proposed architecture for the ingestion pipeline would be:

• S3 trigger attached to the /delivery subfolder of the S3 bucket that writes the key of each new file to an SQS Queue
• The SQS Queue invokes a processing lambda

Some advantages of this architecture:

• It's easy to stop the ingestion pipeline from running by removing the SQS event source from the lambda function (1 line to comment) and re-deploying the CDK stack. In case, any delivered files will be added to the queue, and will stay there until the event source is re-attached to the lambda at which point the ingestion will be back-filled.
• SQS will scale up the number of simultaneous lambda invocations, as long as there are no error statuses being returned by the processing lambda, and there is available lambda concurrency, meaning that the data ingestion pipeline won't consume lambda concurrency that is otherwise needed for other applications (since data ingestion doesn't have to be immediate).

Potential improvements to the architecture:

• The SQS records may contain several (probably up to a dozen or so) records of newly created files. In the case of CMIP6 daily data, for example, 12 files represented ~4400 COG's to generate, which was sometimes exceeding the lambda's 15 min timeout. A way to mitigate this would be to implement a "fan-out" lambda, whose job is to invoke the processing lambda over each.
• A second way to mitigate the above issue is to have the "fan-out" logic orchestrated by a stepfunction

A note on repository structure:

Q: Should all three backend services (tiling API, STAC API and ingest pipeline) be their own separate repositories? A: The fact that this issue is in a repo called delta-backend seems to indicate that we've answered no to this question, but I'm not convinced. The answer to this question is also affected by the decision of how we implement the tiling API and STAC API (ie: are both separate FastAPI apps? Is the TiTiler added as an extension to stac-fastapi? Does titiler-pgstac provide a STAC API endpoint?

As we answer these questions, we should have a clearer idea as to how to structure the backend (1 vs 3 github repos). Some things to keep in mind (or to debate):

• Users should be able to deploy the entire backend with a single CDK deploy command
• If users don't need a STAC API they shouldn't have to fork a repo containing one and have to deploy it
• Since users will have to fork the delta-config repo anyways, in order to be able to set up their own instance of the delta-dashboard, any further customization or configuration should happen through that repo, as much as possible. Ideally we would want to avoid a situation where the user has to fork addition repositories in addition to the delta-config

A note on CDK structure:

Following some of these articles:

• https://docs.aws.amazon.com/cdk/latest/guide/best-practices.html
• https://aws.amazon.com/blogs/developer/recommended-aws-cdk-project-structure-for-python-applications/ on structuring CDK applications, I'd like to implement the following:

CDK stacks are composable (ie: a single stack can be made up of other stacks). I would like the backend to be deployed as a single stack, but composed of sub-stacks, that are each individually deployed on their as well. Arbitrarily assuming that all three component stay in the same repo for the sake of this example, an example file structure would look like:

tiler_api/
  |_ setup.py  # runtime libs + CDK libs for needed to deploy   
  |_ stack.py # CDK Stack for the TiTiler app
  |_ runtime/
      |_ api/
        |_ main.py # fastapi app definition that imports TiTiler and adds it to the routes    
        |_... # other fastapi related files
stac_api/
  |_ setup.py # runtime libs  + CDK libs for needed to deploy
  |_ stack.py # CDK Stack for the STAC API
  |_ db/
    |_ construct.py # CDK file that generates a Construct for the RDS instance
  |_ bootstrapper/
    |_ infrastructure/ 
      |_construct.py # CDK file that generates a Construct for the Lambda function that bootstraps the postgres DB
      |_ Dockerfile 
    |_ runtime/
      |_ handler.py
ingest_pipeline/
  |_ setup.py # runtime libs + CDK libs for needed to deploy
  |_ stack.py # CDK Stack for the ingest pipeline (SQS Queue + DeadLetter Queue + S3 Bucket + Trigger)
  |_ processing_lambda/
    |_ runtime/
      |_ handler.py
      |_ ... # other supporting files for processing lambda
  |_ infrastructure/
    |_ Dockerfile
    |_ construct.py
app_stack.py

The main aspects are:

• app_stack.py generates a single app composed of each of the constituent stacks, in order to deploy the entire backend with a single CDK deploy
• Each component (tiler API, STAC API, ingest pipeline) has it's own stack.py, which enables the deployment of that component on its own
• If the component (tiler API, STAC API, ingest pipeline) is simple enough, it only needs the stack.py and a runtime/ folder for any runtime code
• If the component is more complex, it is broken up into sub-components. Each sub-components has file called construct.py that exposes a construct for that sub-component. If the sub-component requires runtime code (such as a lambda function), that goes into a folder called runtime/ . If the component's construct requires external resources (such as a Dockerfile) then construct.py file gets saved in the infrastructure/ sub-folder, along with any other files needed for the construct.

cc/ @anayeaye @abarciauskas-bgse @olafveerman @drewbo

[vincentsarago]

👋 @leothomas First great ticket 👏

Tiling API:

Option 1 Is def the best and why we built titiler as python module.

Stac-API

Is there an out-of-the-box integration with TiTiler somehow?

There was, but it has been removed because both services shouldn't run in the same application.

TiTiler-pgstac:

Does this extension provide a STAC API endpoint on the tiler? Or just the ability to generate mosaic's from STAC queries?

No, again stac-api and tiler application should always be separated.

eoAPI

Must of what you are describing (api + deployment) is what I've been experiencing with https://github.com/developmentseed/eoAPI

It's a repo with both Tiler and STAC-API python application + CDK code. (note: it's not up to date)

Note:

in eoAPI we have added a hack to link the stac-api and titiler.

we have added TiTilerExtension which adds a /collections/{collectionId}/items/{itemId}/tilejson.json" endpoint https://github.com/developmentseed/eoAPI/blob/master/src/eoapi/stac/eoapi/stac/extension.py#L22 which itself redirect to titiler endpoints (stac-api deployment needs to know the titiler endpoint url)

BUT the real hack is that we are passing the STAC item with the url (this is to avoid for the tiler to fetch the stac item on each tiler requests) https://github.com/developmentseed/eoAPI/blob/master/src/eoapi/stac/eoapi/stac/extension.py#L100

Happy to talk more about eoAPI if you need :-)

[anayeaye]

Sharing this sketch discuss what I think are our current high level architecture choices (and questions, and probably some of my own misunderstandings). With the examples in eoAPI, I think we can start building out piecewise from the RDS construct.

Base app:

• VPC

Initial constructs:

1. PgSTAC [RDS]
2. TiTiler API (pip install until/if customization needed) [ECS]
3. STAC-Fast API (pip install until customization needed) [ECS]

(source)

[leothomas]

@anayeaye This diagram is super helpful, thanks!

I've got a couple questions/comments that I'd appreciate your input on:

• Since we're going to deploy the STAC API and the Tiling API separately, it might be a good idea to deploy the Tiling API as an ECS service and the STAC API as a lambda function (as a "best of worlds" situation)? I may be wrong, but I'm assuming the STAC API just does DB lookups, so there less concern about very memory intensive, long running operations, and file-cacheing in GDAL
• Decoupling the Tiling API from the STAC Database (ie: TiTiler only communicates with STAC API through HTTP Requests and then STAC API communicates with the RDS instance) might make the project easier to deploy the Tiling API on top of an existing STAC API (where the person deploying the Tiling API might not even have access to the RDS isntance)

[anayeaye]

@leothomas I hastily updated the sketch after our tag-up with @vincentsarago to make it a little less wrong. This diagrammed scenario is for a stack that runs two services (A stac-api and a tiling-api) for same RDS PgSTAC instance. I don't think we settled on this design but I wanted to try to capture the scenario anyway.

CHANGED:

1. Separate of tiler and stac-api (connection in original drawing was wrong)
   • STAC-FastAPI is only for metadata requests and possibly transactions to load ingested metadata. Or maybe we prefer to use pypgstac to load metadata in database and do not enable transactions in STAC-FastAPI because we don't want to expose edit operations to all API users.
   • TiTiler-PgSTAC API accepts a STAC search query but it connects directly to the database (does not use STAC-FastAPI)
2. Use Lambda infrastructure for more responsive scaling.
3. Placeholders for CloudFront(s) and Auth that I think we will want but that needs thought.

EDIT: (source)

[abarciauskas-bgse]

I second @vincentsarago comment that this is a great ticket @leothomas

Exciting to hear about eoAPI, I think we will be using that a lot in this and other projects 😃

And thanks for the great diagrams @anayeaye