Idea/Proposal: EDGE-URID

[alvin-reyes]

Edge-urid

Functional / features

• Content creator
• File aggregator
  • Aggregates small files (> BUCKET_AGG_SIZE) to bucket
  • The last file to be added can exceed the BUCKET_AGG_SIZE
  • Creates a CAR file for the bucket
• File splitter
  • Splits a large file into (SPLIT_SIZE configuration)
  • Applies to files that are larger than the BUCKET_AGG_SIZE.
  • The large file will be associated to a new bucket with each split being added as a content
  • Creates a CAR file for the bucket
• Bucket-creator
  • If the content has a miner, a bucket will be created for the miner and for the content. This bucket will need to be filled based on BUCKET_AGG_SIZE.
• Deal-checker
  • If bucket is already replicated more than (MAX_REP) times, it can be deleted by the ADMIN
• Gateway for serving content

System objects

• Content
• Content_Deals
• Buckets

Endpoints

To serve file

• gw/

To add a file

• add
  • File, bucket_uuid (optional), miner (optional)
• gw
  • File, bucket_uuid (optional), miner (optional)
• fetch-from-url
  • File, bucket_uuid (optional), miner (optional)

To get buckets (pull model from SP)

• get-available-open-buckets
  • Return list of CAR files and COMMP and deal request metadata
• Get-open-bucket
  • Return the CAR file and COMMP and deal request metadata
• get-private-bucket (private)
  • Return list of CAR files and COMMP of each for a specific bucket

To manage buckets

• create-bucket (admin or anyone)
  • create a bucket with specific meta/keys
  • bucket_uuid
  • miner
• delete-buckets

To get status

• status/content
• status/bucket
• status/cid

To get stats

• stats
  • Number of content
  • Number of buckets
  • Number of deals attempted
  • Number of deals made

[jcace]

From our chat today @alvin-reyes , quick outline of the tag idea:

• Admin endpoints to create and delete tags (+ modify ? )
• When user uploads content, they can optionally specify a tag. If unspecified, uses default tag
• tag is where we specify the BUCKET_AGG_SIZE and SPLIT_SIZE parameters
• buckets are a lower level abstraction,tag is also automatically to the resulting bucket where contents end up
• a single bucket will never mix different tag s -> resulting carfiles will only contain contents with the same tag
• Thus, tag:bucket is 1:many , tag:content is also 1:many
• When user calls get-available-open-buckets, return the tag param for each bucket too

Open questions:

• Not entirely sure how to handle the delete case. If a tag has already generated a bunch of carfiles then what happens to them?
• Allowing tag renaming would also be nice too. Easy to do if tag are indexed by ID. Just need to make sure we use the id when referencing from DDM.
• Could a tag's BUCKET_AGG_SIZE and SPLIT_SIZE be modified after it's set?