search

developmentseed / eoAPI

[Active Development] Earth Observation API (Metadata, Raster and Vector services)
https://eoapi.dev
MIT License
196 stars 21 forks source link

Transition eoAPI infrastructure to external repositories #94

Closed sharkinsspatial closed 9 months ago

sharkinsspatial commented 1 year ago

The libraries that eoAPI demonstrates are the same ones that we already use as the foundation for many of our partner projects (both with NASA and AWS). We've had several internal discussions about how we can potentially change the structure of eoAPI to both show that this combination of libraries is a robust stack that is used in production systems and make this combination of technologies easy for other teams to deploy and customize for their projects. To do this we want to

  1. Capture the infrastructure best practices and patterns we've learned from our production systems in reusable IAC that other teams can leverage and customize when needed.

  2. Support cloud providers other than AWS through a complete k8s IAC solution.

  3. Deploy the APIs with sensible defaults that support the most commonly required basic functionality.

  4. Allow @vincentsarago to continue to use the eoAPI repo as demonstration of the newest cutting edge features in upstream libraries while also providing users more stable deployments with fully tested releases.

In our previous discussions around this I had focused on us transitioning eoAPI to a monorepo that included multiple deployments to various package managers to support IAC package releases. My rationale for this was that we've essentially copy and pasted the eoAPI runtime code for several of our projects and I thought it might be cleaner for those projects to simply use a centralized package. After chatting with @vincentsarago yesterday and noting how thin the runtime code is, we've decided on the opposite approach. We'll focus on moving IAC to external repos (with their own default runtime copies) and the eoAPI can use the AWS CDK IAC library (with runtime overrides) for deployments of its local demonstration runtimes. To tackle this we can

  1. Create a public CDK stack repo (using our networking best practices) that uses the cdk-pgstac construct. This stack should be fully self-contained and allow users to deploy with their own configuration. We have a complete example of this in an existing private DevSeed repo.

  2. Remove the current eoAPI infrastructure in favor of using the CDK stack repo from above and override its default runtimes with the demonstration runtimes in eoAPI (this is already supported in cdk-pgstac).

  3. Move @ranchodeluxe 's spike work on k8s infrastructure to a standalone repo that includes its own default runtimes (and the CI to deploy them as tagged images in public repository). Like the CDK stack, this k8s application should be fully self-contained and allow users to deploy with their own configuration.

  4. Update the landing page documentation to describe "eoAPI" as this suite of core libraries (stac-fastapi, pgSTAC, titiler-pgSTAC) that we use to build applications and point to the external IAC repositories as the place users can go to get standalone deployment packages with a regular release cadence. Differentiate the fact that the runtimes in the eoAPI repository are to demonstrate the newest features for experimentation.

These tasks should provide a good initial target. As discussed on our internal Slack, one of the current pain points with the eoAPI infrastructure is the difficulty in loading sample data for testing due to network security constraints. To remedy this and also support large scale data ingestion in both AWS and non-AWS providers we should

  1. Support the transaction extension in cdk-pgstac and have it enabled with simple security when used in the eoAPI demonstration deployment. This way users can easily load experimental data through a REST endpoint without additional library requirements or networking knowledge.

  2. cdk-pgstac already supports a large scale data ingestion solution, but it is reliant on AWS native services. We'll also need a separate effort to create an agnostic large scale ingestion solution for our k8s deployments.

vincentsarago commented 1 year ago

zacharyDez commented 10 months ago

@sharkinsspatial; what is missing for this to be closed? It seems like we made significant progress, but there may be some details remaining.

sharkinsspatial commented 10 months ago

@zacharyDez We still need to transition this repo to use the eoapi-cdk constructs (with the local runtimes injected) for it's AWS deployment which @emileten and @vincentsarago are working on https://github.com/developmentseed/eoAPI/pull/144 and then I think we will be good to close this. I'm going to make another issue directly on eoapi-k8s around next steps for a large scale ingestion solution we can use for k8s.

emileten commented 9 months ago

I am closing this since we just merged https://github.com/developmentseed/eoAPI/pull/144.