⁂
web3.storage
The filecoin pipeline infra for [w3protocol] built on [SST]. w3filecoin is responsible for keeping the state over time from CAR files being ready to aggregate, to an aggregate landing in Filecoin Storage providers.
## Getting Started
The repo contains the infra deployment code and the service implementation.
```
├── stacks - sst and aws cdk code to deploy all the things
└── api - lambda & dynamoDB implementations for the filecoin pipeline
```
To work on this codebase **you need**:
- Node.js >= v18 (prod env is node v18)
- An AWS account with the AWS CLI configured locally
- Copy `.env.tpl` to `.env.local`
- Install the deps with `npm i`
Deploy dev services to your aws account and start dev console
```console
npm start
```
See: https://docs.sst.dev for more info on how things get deployed.
## Deployment
Deployment is managed by [seed.run]. PR's are deployed automatically to `https://.filecoin.web3.storage`.
The `main` branch is deployed to https://staging.filecoin.web3.storage and staging builds are promoted to prod manually via the UI at https://console.seed.run
### Local dev
You can use `sst` to create a custom dev deployment on aws, with a local dev console for debugging.
To do that **you need**
- An AWS account with the AWS CLI configured locally
- Copy `.env.tpl` to `.env.local`
Then run `npm dev` to deploy dev services to your aws account and start dev console
See: https://docs.sst.dev for more info on how things get deployed.
### Environment Variables
Ensure the following variables are set in the env when deploying
#### `AGGREGATOR_HOSTED_ZONE`
The root domain to deploy the aggregator API to. e.g `aggregator.web3.storage`. The value should match a hosted zone configured in route53 that your aws account has access to.
#### `DEAL_TRACKER_API_HOSTED_ZONE`
The root domain to deploy the deal tracker API to. e.g `tracker.web3.storage`. The value should match a hosted zone configured in route53 that your aws account has access to.
#### `DEALER_API_HOSTED_ZONE`
The root domain to deploy the dealer API to. e.g `dealer.web3.storage`. The value should match a hosted zone configured in route53 that your aws account has access to.
#### `AGGREGATOR_DID`
[DID](https://www.w3.org/TR/did-core/) of the ucanto server running for the Aggregator service. e.g. `did:key:abc..`. Optional: if omitted, a `did:key` will be derrived from `AGGREGATOR_PRIVATE_KEY`
#### `DEAL_TRACKER_DID`
[DID](https://www.w3.org/TR/did-core/) of the ucanto server running for the Deal Tracker service. e.g. `did:key:abc..`. Optional: if omitted, a `did:key` will be derrived from `DEAL_TRACKER_PRIVATE_KEY`
#### `DEALER_DID`
[DID](https://www.w3.org/TR/did-core/) of the ucanto server running a Dealer service. e.g. `did:web:dealer.web3.storage`.
#### `UCAN_LOG_URL`
URL of the UCAN log server. e.g. `https://up.web3.storage/ucan`
### Secrets
Set production secrets in aws SSM via [`sst secrets`](https://docs.sst.dev/config#sst-secrets). The region must be set to the one you deploy that stage to
```sh
# set `PRIVATE_KEY` for prod
$ npx sst secrets set --region us-west-2 --stage prod PRIVATE_KEY "MgCblCY...="
```
To set a fallback value for `staging` or an ephmeral PR build use [`sst secrets set-fallback`](https://docs.sst.dev/config#fallback-values)
```sh
# set `PRIVATE_KEY` for any stage in us-east-2
$ npx sst secrets set --fallback --region us-east-2 PRIVATE_KEY "MgCZG7...="
```
**note** The fallback value can only be inherited by stages deployed in the same AWS account and region.
Confirm the secret value using [`sst secrets list`](https://docs.sst.dev/config#sst-secrets)
```sh
$ npx sst secrets list --region us-east-2
PRIVATE_KEY MgCZG7...= (fallback)
$ npx sst secrets list --region us-west-2 --stage prod
PRIVATE_KEY M...=
```
#### `AGGREGATOR_PRIVATE_KEY`
The `base64pad` [`multibase`](https://github.com/multiformats/multibase) encoded ED25519 keypair used as the signing key for the aggregator-api.
Generated by [@ucanto/principal `EdSigner`](https://github.com/web3-storage/ucanto) via [`ucan-key`](https://www.npmjs.com/package/ucan-key)
_Example:_ `MgCZG7EvaA...1pX9as=`
#### `DEALER_PRIVATE_KEY`
The `base64pad` [`multibase`](https://github.com/multiformats/multibase) encoded ED25519 keypair used as the signing key for the dealer-api.
Generated by [@ucanto/principal `EdSigner`](https://github.com/web3-storage/ucanto) via [`ucan-key`](https://www.npmjs.com/package/ucan-key)
_Example:_ `MgCZG7EvaA...1pX9as=`
#### `DEAL_TRACKER_PRIVATE_KEY`
The [`multibase`](https://github.com/multiformats/multibase) encoded ED25519 keypair used as the signing key for the deal-tracker-api.
Generated by [@ucanto/principal `EdSigner`](https://github.com/web3-storage/ucanto) via [`ucan-key`](https://www.npmjs.com/package/ucan-key)
_Example:_ `MgCZG7EvaA...1pX9as=`
#### `UCAN_LOG_BASIC_AUTH`
The HTTP Basic auth token for the UCAN Invocation entrypoint, where UCAN invocations can be stored and proxied to the UCAN Stream.
_Example:_ `MgCZG7EvaA...1pX9as=`