search

storacha-network / w3infra

🏗️ Infra for the w3up UCAN protocol implementation
Other
15 stars 5 forks source link

feat: add blob protocol to infra #353

Closed vasco-santos closed 4 months ago

vasco-santos commented 4 months ago

This PR creates stores and wires up new upload-api running blob/add, web3.storage/blob/allocate, web3.storage/blob/accept and ucan/conclude capabilities. Tests are also imported from upload-api implementation and run here.

As agreed on https://github.com/web3-storage/w3up/issues/1343 , there won't be any deduping between old world and new world. Therefore, we have new allocations table, and use different key schema in carpark. We are writing blobs keyed as base58btc as previously discussed as ${base58btcEncodedMultihash}/${base58btcEncodedMultihash}.blob. I added .blob suffix but I am happy to other suggestions. Depending on how we progress with the reads side, we can consider creating a new bucket to fully isolate new content?

The receipts and tasks storage end up being more complicated as they need to follow https://github.com/web3-storage/w3infra/blob/main/docs/ucan-invocation-stream.md#buckets, and is essentially the same as what happens on https://github.com/web3-storage/w3infra/blob/main/upload-api/ucan-invocation.js#L66 but at a different level as this is a proactive write of tasks and receipts.

Integration tests are in progress in branch https://github.com/web3-storage/w3infra/tree/feat/add-blob-protocol-to-service-integration-test. They currently run in my local dev setup, and have full functionality from blob/add, followed by HTTP PUT to bucket and ucan/conclude containing the receipt of the put. This makes scheduled blob/accept run and a receipt with a location claim being available. You can see the tests in branch, on file test/blob.test.js. I want to only add them here once we can rely on the client to do the manual hack I did here

seed-deploy[bot] commented 4 months ago
View stack outputs - **pr353-w3infra-BillingDbStack** Name | Value -- | -- customerTableName | pr353-w3infra-customer spaceDiffTableName | pr353-w3infra-space-diff spaceSnapshotTableName | pr353-w3infra-space-snapshot usageTable | pr353-w3infra-usage - **pr353-w3infra-BillingStack** Name | Value -- | -- ApiEndpoint | https://6qn3gn2pa1.execute-api.us-east-2.amazonaws.com billingCronHandlerURL | https://4dwn4epwfqj7nwfjamlhndkoai0csuiq.lambda-url.us-east-2.on.aws/ CustomDomain | https://pr353.billing.web3.storage - **pr353-w3infra-CarparkStack** Name | Value -- | -- BucketName | carpark-pr353-0 Region | us-east-2 - **pr353-w3infra-RoundaboutStack** Name | Value -- | -- ApiEndpoint | https://2ofcus8178.execute-api.us-east-2.amazonaws.com CustomDomain | https://pr353.roundabout.web3.storage - **pr353-w3infra-SatnavStack** Name | Value -- | -- BucketName | satnav-pr353-0 Region | us-east-2 - **pr353-w3infra-UploadApiStack** Name | Value -- | -- ApiEndpoint | https://5qptdfblx9.execute-api.us-east-2.amazonaws.com CustomDomain | https://pr353.up.web3.storage - **pr353-w3infra-BusStack** - **pr353-w3infra-FilecoinStack** - **pr353-w3infra-ReplicatorStack** - **pr353-w3infra-UcanFirehoseStack** - **pr353-w3infra-UcanInvocationStack** - **pr353-w3infra-UploadDbStack**