search

storacha-network / w3infra

🏗️ Infra for the w3up UCAN protocol implementation
Other
13 stars 5 forks source link

feat: add blob capabilities to ucan firehose #375

Closed vasco-santos closed 1 month ago

vasco-santos commented 2 months ago

Adds blob capabilities to UCAN Firehose, and adds glue tables for the main capabilities.

See stream log store for formats: https://us-east-2.console.aws.amazon.com/s3/buckets/stream-log-store-pr375-0?region=us-east-2&bucketType=general&tab=objects

See glue tables:

See query editor: https://us-east-2.console.aws.amazon.com/athena/home?region=us-east-2#/query-editor/history/7a07d617-5c9f-488f-8d86-5d5ce75340fa

You can do queries like: SELECT * FROM "pr375-w3infra-ucan-stream-delivery-database-0"."pr375-w3infra-blob-add-table-0" limit 10; to see the outputs

Notes:

  1. This is dependent on the capabilities name changes to avoid breaking change
  2. I did not change the saved queries yet. Some may be difficult to change to use both store/add and blob/add given nb types are different (e.g. nb.size in store/add and nb.blob.size in blob/add), so I would much rather try to identify what we need as part of the metrics work @travis has been on
seed-deploy[bot] commented 2 months ago
View stack outputs - **pr375-w3infra-BillingDbStack** Name | Value -- | -- customerTableName | pr375-w3infra-customer spaceDiffTableName | pr375-w3infra-space-diff spaceSnapshotTableName | pr375-w3infra-space-snapshot usageTable | pr375-w3infra-usage - **pr375-w3infra-BillingStack** Name | Value -- | -- ApiEndpoint | https://qxbp706x3l.execute-api.us-east-2.amazonaws.com billingCronHandlerURL | https://b3ijg67fliqjgub3ensybzeb4a0gmytj.lambda-url.us-east-2.on.aws/ CustomDomain | https://pr375.billing.web3.storage - **pr375-w3infra-CarparkStack** Name | Value -- | -- BucketName | carpark-pr375-0 Region | us-east-2 - **pr375-w3infra-RoundaboutStack** Name | Value -- | -- ApiEndpoint | https://dllbzi32kd.execute-api.us-east-2.amazonaws.com CustomDomain | https://pr375.roundabout.web3.storage - **pr375-w3infra-SatnavStack** Name | Value -- | -- BucketName | satnav-pr375-0 Region | us-east-2 - **pr375-w3infra-UploadApiStack** Name | Value -- | -- ApiEndpoint | https://zxfzkjlm72.execute-api.us-east-2.amazonaws.com CustomDomain | https://pr375.up.web3.storage - **pr375-w3infra-BusStack** - **pr375-w3infra-FilecoinStack** - **pr375-w3infra-ReplicatorStack** - **pr375-w3infra-UcanFirehoseStack** - **pr375-w3infra-UcanInvocationStack** - **pr375-w3infra-UploadDbStack**
seed-deploy[bot] commented 1 month ago
Stack outputs updated - **pr375-w3infra-BillingDbStack** Name | Value -- | -- customerTableName | pr375-w3infra-customer spaceDiffTableName | pr375-w3infra-space-diff spaceSnapshotTableName | pr375-w3infra-space-snapshot usageTable | pr375-w3infra-usage - **pr375-w3infra-BillingStack** Name | Value -- | -- ApiEndpoint | https://qxbp706x3l.execute-api.us-east-2.amazonaws.com billingCronHandlerURL | https://b3ijg67fliqjgub3ensybzeb4a0gmytj.lambda-url.us-east-2.on.aws/ CustomDomain | https://pr375.billing.web3.storage - **pr375-w3infra-CarparkStack** Name | Value -- | -- BucketName | carpark-pr375-0 Region | us-east-2 - **pr375-w3infra-RoundaboutStack** Name | Value -- | -- ApiEndpoint | https://dllbzi32kd.execute-api.us-east-2.amazonaws.com CustomDomain | https://pr375.roundabout.web3.storage - **pr375-w3infra-UcanInvocationStack** Name | Value -- | -- invocationBucketName | invocation-store-pr375-0 taskBucketName | task-store-pr375-0 workflowBucketName | workflow-store-pr375-0 - **pr375-w3infra-UploadApiStack** Name | Value -- | -- ApiEndpoint | https://zxfzkjlm72.execute-api.us-east-2.amazonaws.com CustomDomain | https://pr375.up.web3.storage - **pr375-w3infra-BusStack** - **pr375-w3infra-FilecoinStack** - **pr375-w3infra-ReplicatorStack** - **pr375-w3infra-UcanFirehoseStack** - **pr375-w3infra-UploadDbStack**