storacha-network / w3infra

🏗️ Infra for the w3up UCAN protocol implementation
Other
15 stars 5 forks source link

feat: refactor kinesis upload api metrics consumers to one for admin and one for space #315

Closed vasco-santos closed 8 months ago

vasco-santos commented 8 months ago

Main motivation for the refactor is the current deployment issues of having too many Kinesis consumers in prod (while in theory there is no limit, on creating resources there are rate limits that make cloudformation fail with SST, perhaps a old version issue too...)

Per convo with Alan, we decided to just merge these Kinesis consumers similarly to how filecoin and billing are using it, i.e. use Kinesis stream resource but own the metrics. So, upload-api now owns lambdas and tables for its metrics by hooking into kinesis streams.

To avoid getting into limits with batch write to DB, while easing logic for single DB transaction per lambda, they are still two kinesis consumers instead of getting to one. So one per table: Admin Metrics, and Space Metrics

Ended up fully removing ucan-invocation package as it became useless and will make it easier to transition the ucan stream into a single package once we work on upgrading SST here :)

seed-deploy[bot] commented 8 months ago
View stack outputs - **pr315-w3infra-BillingDbStack** Name | Value -- | -- customerTableName | pr315-w3infra-customer spaceDiffTableName | pr315-w3infra-space-diff spaceSnapshotTableName | pr315-w3infra-space-snapshot usageTable | pr315-w3infra-usage - **pr315-w3infra-BillingStack** Name | Value -- | -- ApiEndpoint | https://nkmay45tva.execute-api.us-east-2.amazonaws.com billingCronHandlerURL | https://g6j5w5o74bfgjt5xzfstq4wbx40cvhzx.lambda-url.us-east-2.on.aws/ CustomDomain | https://pr315.billing.web3.storage - **pr315-w3infra-CarparkStack** Name | Value -- | -- BucketName | carpark-pr315-0 Region | us-east-2 - **pr315-w3infra-RoundaboutStack** Name | Value -- | -- ApiEndpoint | https://1v87qc98za.execute-api.us-east-2.amazonaws.com CustomDomain | https://pr315.roundabout.web3.storage - **pr315-w3infra-SatnavStack** Name | Value -- | -- BucketName | satnav-pr315-0 Region | us-east-2 - **pr315-w3infra-UploadApiStack** Name | Value -- | -- ApiEndpoint | https://0h300y4dz9.execute-api.us-east-2.amazonaws.com CustomDomain | https://pr315.up.web3.storage - **pr315-w3infra-BusStack** - **pr315-w3infra-FilecoinStack** - **pr315-w3infra-ReplicatorStack** - **pr315-w3infra-UcanFirehoseStack** - **pr315-w3infra-UcanInvocationStack** - **pr315-w3infra-UploadDbStack**
sentry-io[bot] commented 8 months ago

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

Did you find this useful? React with a 👍 or 👎