Tracking: PoC checklist

[hopeyen]

This issue should track all the items needed for a proof of concept. Aim to finish by Dec 1st, basic testing done by Dec 20.

Start with firehose flatfiles, data verifiability guaranteed by ovc and files decoder

More updated details can be found in README.md under the PoC checklist section.

General

Minimal components

• [ ] Standarlise subfile.yaml manifest formats (in subfile_manifest.md)
• [x] Basic indexer selection algorithm done in client CLI (data availability checked with each indexer endpoints)
• [ ] Draft out GRC
• [x] basic documentations, usage, and architectural diagrams

Next steps

• verifiable TAP payments on data chunks
• verifiable committed blockchain data
• verifiable staking for computed blockchain data
• More intelligent indexer selection algorithms - Economics research
• Subfile data service enabled on the staking contract

Publisher/Provider

We can expect that the provider will use an CLI to interact with the continuous service. They can create a subfile service (as a deployment unit) on-chain and/or start serving the file off-chain. We assume that the the file is accessible by the continuous service and not necessarily by the CLI.

Minimal components

• [x] CLI: Create a subfile and publish to IPFS
• [x] CLI: Start serving a particular subfile by IPFS hash
• [x] Service: Receive request from CLI command at admin API and do corresponding actions
• [x] Service: host a file availability endpoint (/status)
• [x] Service: start with free service for a single file, (/subfiles/:id)
  • [x] continuously host the subfile
  • [x] require a free_query_auth_token for service
  • [x] take partial download request, verify validity of the request (valid range)
  • [x] grab the chunk by the range and respond
  • [x] support hosting multiple subfiles
• [x] CLI: Delete subfile from service

Next steps

• Paid query flow: parse, validate, store, and redeem receipts with TAP
• cost models: store a db of cost models for each torrent file according to chunk sizes, serve a dedicated route on subfile-service

Client

Clearly state the limitations of our approach that while payment is minimized to a chunk at a time, We require 1 of n (provider) trust as there is no guarantee for a subfile to be completely torrented if no indexer serves the target files.

Minimal components

Assume the client is capable of identifying the correct ipfs hash and maintain budget balance on-chain

• [x] CLI takes in request (ipfs_hash)
  • [x] Indexer selection - This may live somewhere else
    • [x] Start with an static indexer status endpoint, with a free query token
    • [x] Ping indexer endpoints for availability
    • [x] Construct and send requests (must be parallelizable) to indexer endpoints
  • [x] Wait for the responses (For now, assume that the response chunks correspond with the verifiable chunks)
  • [x] Keeps track of the downloaded and missing pieces
  • [x] Multiple attempts for requesting missing pieces
  • [x] Upon receiving a response, verify the chunk data in the chunk_file
    • [x] if failed, blacklist the indexer
  • [x] Once all chunks for a file has been received, verify the file in subfile (should be vacuously true)
  • [x] Once all file has been received and verified, terminate client

Next steps

• client: expose an endpoint for showing download progress
• CLI: stop download by subfile hash
• Payment: Read subfile manifest and construct receipts using budget and chunk sizes
• Payment: build and send TAP receipts

Testing and Documentation

• [x] Conduct basic testing to ensure functionality, reliability, and security
• [x] Benchmark performant sensitive functions
• [x] Create foundational documentation outlining the usage, architecture, and known limitations.

Goals of MVP

• Validate the feasibility and utility of the data exchange service.
• Gather early feedback from users and identify areas for improvement.
• Identify unforeseen challenges or limitations that arise during development.

Scope outside of MVP

• Advanced dispute resolution mechanisms: this may live on Horizon
• Optimal service and price matching algorithms: conduct Economics research on information markets, building on top of this paper
• Formal verifications on data validation and integrity checks.
• Detailed and polished user interface.

Post-MVP Developments

Once the MVP is successfully developed, tested, and validated, subsequent iterations would focus on

• refining the existing features,
• adding the excluded advanced features,
• optimizing performance,
• enhancing security,
• and improving user experience based on feedback and requirements.

[hopeyen]

Moving the leftover 2 tasks to MVP checklist, marking done for POC stage

Standarlise subfile.yaml manifest formats (in subfile_manifest.md) Draft out GRC