search

storacha / w3infra

🏗️ Infra for the w3up UCAN protocol implementation
Other
16 stars 6 forks source link

feat: partition kinesis messages by task CID #408

Closed alanshaw closed 2 months ago

alanshaw commented 2 months ago

This PR adds a partition key to kinesis messages that allows consumers to consume messages in parallel using parallelizationFactor.

Context: We added parallelizationFactor but it had no effect. This is because we set the same partition key for every message.

seed-deploy[bot] commented 2 months ago
View stack outputs - **pr408-w3infra-BillingDbStack** Name | Value -- | -- customerTableName | pr408-w3infra-customer spaceDiffTableName | pr408-w3infra-space-diff spaceSnapshotTableName | pr408-w3infra-space-snapshot usageTable | pr408-w3infra-usage - **pr408-w3infra-BillingStack** Name | Value -- | -- ApiEndpoint | https://6mulnsl890.execute-api.us-east-2.amazonaws.com billingCronHandlerURL | https://6zgqirfs52l5ifcyabldfimjiy0xwupx.lambda-url.us-east-2.on.aws/ CustomDomain | https://pr408.billing.web3.storage - **pr408-w3infra-CarparkStack** Name | Value -- | -- BucketName | carpark-pr408-0 Region | us-east-2 - **pr408-w3infra-RoundaboutStack** Name | Value -- | -- ApiEndpoint | https://f1w6mv7td3.execute-api.us-east-2.amazonaws.com CustomDomain | https://pr408.roundabout.web3.storage - **pr408-w3infra-UcanInvocationStack** Name | Value -- | -- invocationBucketName | invocation-store-pr408-0 taskBucketName | task-store-pr408-0 workflowBucketName | workflow-store-pr408-0 - **pr408-w3infra-UploadApiStack** Name | Value -- | -- ApiEndpoint | https://ahrkqsar8f.execute-api.us-east-2.amazonaws.com CustomDomain | https://pr408.up.web3.storage - **pr408-w3infra-BusStack** - **pr408-w3infra-FilecoinStack** - **pr408-w3infra-IndexerStack** - **pr408-w3infra-ReplicatorStack** - **pr408-w3infra-UcanFirehoseStack** - **pr408-w3infra-UploadDbStack**
alanshaw commented 2 months ago

Going to merge this so we can make progress.