Expose issues in Parquet format via pipeline API

[Ben-Hodgkiss]

Overview

Expose issues in Parquet format via new pipeline API

Background Following the design proposal for an internal API, we would like to prove some technology choices which include the use of Fast API with DuckDB accessing Parquet on S3.

This work was identified during the spike on API design.

By exposing the issues in Parquet format via an internal API, we will know whether Parquet will be an appropriate format for consuming via an internal API as well as datasette.

Builds on work to be delivered via https://trello.com/c/NZevJ4wU/3581-spike-prove-fast-api-with-duckdb-and-parquet-on-s3 .

Tech Approach

• Build on Fast API implementation for Pipeline API by adding unit and integration tests
• Add GHA CI workflow to publish docker image
• Provision AWS resources to run pipeline API in dev, staging and prod

Acceptance Criteria/Tests

• Code merged within pipeline-api repo
• Pipeline API exposing issues in Parquet format deployed to production environment
• Integrate with Sentry

---

Ticket Management - DELETE this section once completed

• Complete all relevant tags - make sure Infrastructure is tagged so it is picked up by our filters!
• Complete the time estimate field
• Make sure you have a PR link in the Overviewabove.
• If relevant, link to the relevant OKR as an attachment.
• Link to any tickets in other boards that are dependent on it.