storacha / w3infra

🏗️ Infra for the w3up UCAN protocol implementation
Other
17 stars 7 forks source link

feat: use batching for billing UCAN stream handler #410

Closed alanshaw closed 3 months ago

alanshaw commented 3 months ago

This enables the UCAN stream handler to accept messages in batches, which should hopefully speed up processing of the stream and stop the backlog from growing.

It's ok for the whole batch to be retried on failure since the primary key does not change over time, so any records that succeeded to be written in a failed batch will just be overwritten on retry.

seed-deploy[bot] commented 3 months ago
View stack outputs - **pr410-w3infra-BillingDbStack** Name | Value -- | -- customerTableName | pr410-w3infra-customer spaceDiffTableName | pr410-w3infra-space-diff spaceSnapshotTableName | pr410-w3infra-space-snapshot usageTable | pr410-w3infra-usage - **pr410-w3infra-BillingStack** Name | Value -- | -- ApiEndpoint | https://hnm355s8vj.execute-api.us-east-2.amazonaws.com billingCronHandlerURL | https://mqxfqhhmbqmiqq5dvsuze4ggou0gycqk.lambda-url.us-east-2.on.aws/ CustomDomain | https://pr410.billing.web3.storage - **pr410-w3infra-CarparkStack** Name | Value -- | -- BucketName | carpark-pr410-0 Region | us-east-2 - **pr410-w3infra-RoundaboutStack** Name | Value -- | -- ApiEndpoint | https://xa677qf5c6.execute-api.us-east-2.amazonaws.com CustomDomain | https://pr410.roundabout.web3.storage - **pr410-w3infra-UcanInvocationStack** Name | Value -- | -- invocationBucketName | invocation-store-pr410-0 taskBucketName | task-store-pr410-0 workflowBucketName | workflow-store-pr410-0 - **pr410-w3infra-UploadApiStack** Name | Value -- | -- ApiEndpoint | https://650fvnaap5.execute-api.us-east-2.amazonaws.com CustomDomain | https://pr410.up.web3.storage - **pr410-w3infra-BusStack** - **pr410-w3infra-FilecoinStack** - **pr410-w3infra-IndexerStack** - **pr410-w3infra-ReplicatorStack** - **pr410-w3infra-UcanFirehoseStack** - **pr410-w3infra-UploadDbStack**