Document proposed architecture

HTTPArchive / tech-report-apis

APIs for the HTTP Archive Technology Report

Apache License 2.0

2 stars 0 forks source link

Document proposed architecture #1

Open rviscomi opened 1 year ago

rviscomi commented 1 year ago

Describe how we'd serve data from the API endpoint, extracting from BigQuery and updating at the end of each monthly crawl

rviscomi commented 1 year ago

httparchive.core_web_vitals.technologies is the source of truth and there will be one script for each of the API endpoints that query the data from the table

rviscomi commented 1 year ago

Hook into the automated pipeline to update API data on Firestore on a monthly basis: https://github.com/HTTPArchive/data-pipeline/blob/main/data-pipeline.workflows.yaml

Changing the API code itself will require a manual deployment script

rviscomi commented 1 year ago

API security: as needed implement CORS and/or API keys

rviscomi commented 1 year ago

Firestore

global instance?
lives in the project, serverless service
not like a VM instance
data stored in us-east
possible to put a CDN in front of API gateway

maceto commented 1 year ago

@rviscomi @tunetheweb I changed the region for the project to us-east4, which is the default for the project. Now Firestore database, API gateway, and cloud functions are all in the same region. It should work with the performance.

And looking for information about CDN in front of the API, I found this documentation https://cloud.google.com/api-gateway/docs/multi-region-deployment

I tried to test it but I don't have permission to create all necessary resources.