This issue should track all the items needed for a proof of concept. Aim to finish by Dec 1st, basic testing done by Dec 20.
Start with firehose flatfiles, data verifiability guaranteed by ovc and files decoder
More updated details can be found in README.md under the PoC checklist section.
General
Minimal components
[ ] Standarlise subfile.yaml manifest formats (in subfile_manifest.md)
[x] Basic indexer selection algorithm done in client CLI (data availability checked with each indexer endpoints)
[ ] Draft out GRC
[x] basic documentations, usage, and architectural diagrams
Next steps
verifiable TAP payments on data chunks
verifiable committed blockchain data
verifiable staking for computed blockchain data
More intelligent indexer selection algorithms - Economics research
Subfile data service enabled on the staking contract
Publisher/Provider
We can expect that the provider will use an CLI to interact with the continuous service. They can create a subfile service (as a deployment unit) on-chain and/or start serving the file off-chain. We assume that the the file is accessible by the continuous service and not necessarily by the CLI.
Minimal components
[x] CLI: Create a subfile and publish to IPFS
[x] CLI: Start serving a particular subfile by IPFS hash
[x] Service: Receive request from CLI command at admin API and do corresponding actions
[x] Service: host a file availability endpoint (/status)
[x] Service: start with free service for a single file, (/subfiles/:id)
[x] continuously host the subfile
[x] require a free_query_auth_token for service
[x] take partial download request, verify validity of the request (valid range)
[x] grab the chunk by the range and respond
[x] support hosting multiple subfiles
[x] CLI: Delete subfile from service
Next steps
Paid query flow: parse, validate, store, and redeem receipts with TAP
cost models: store a db of cost models for each torrent file according to chunk sizes, serve a dedicated route on subfile-service
Client
Clearly state the limitations of our approach that while payment is minimized to a chunk at a time, We require 1 of n (provider) trust as there is no guarantee for a subfile to be completely torrented if no indexer serves the target files.
Minimal components
Assume the client is capable of identifying the correct ipfs hash and maintain budget balance on-chain
[x] CLI takes in request (ipfs_hash)
[x] Indexer selection - This may live somewhere else
[x] Start with an static indexer status endpoint, with a free query token
[x] Ping indexer endpoints for availability
[x] Construct and send requests (must be parallelizable) to indexer endpoints
[x] Wait for the responses (For now, assume that the response chunks correspond with the verifiable chunks)
[x] Keeps track of the downloaded and missing pieces
[x] Multiple attempts for requesting missing pieces
[x] Upon receiving a response, verify the chunk data in the chunk_file
[x] if failed, blacklist the indexer
[x] Once all chunks for a file has been received, verify the file in subfile (should be vacuously true)
[x] Once all file has been received and verified, terminate client
Next steps
client: expose an endpoint for showing download progress
CLI: stop download by subfile hash
Payment: Read subfile manifest and construct receipts using budget and chunk sizes
Payment: build and send TAP receipts
Testing and Documentation
[x] Conduct basic testing to ensure functionality, reliability, and security
[x] Benchmark performant sensitive functions
[x] Create foundational documentation outlining the usage, architecture, and known limitations.
Goals of MVP
Validate the feasibility and utility of the data exchange service.
Gather early feedback from users and identify areas for improvement.
Identify unforeseen challenges or limitations that arise during development.
Scope outside of MVP
Advanced dispute resolution mechanisms: this may live on Horizon
Optimal service and price matching algorithms: conduct Economics research on information markets, building on top of this paper
Formal verifications on data validation and integrity checks.
Detailed and polished user interface.
Post-MVP Developments
Once the MVP is successfully developed, tested, and validated, subsequent iterations would focus on
refining the existing features,
adding the excluded advanced features,
optimizing performance,
enhancing security,
and improving user experience based on feedback and requirements.
This issue should track all the items needed for a proof of concept. Aim to finish by Dec 1st, basic testing done by Dec 20.
Start with firehose flatfiles, data verifiability guaranteed by ovc and files decoder
More updated details can be found in README.md under the PoC checklist section.
General
Minimal components
Next steps
Publisher/Provider
We can expect that the provider will use an CLI to interact with the continuous service. They can create a subfile service (as a deployment unit) on-chain and/or start serving the file off-chain. We assume that the the file is accessible by the continuous service and not necessarily by the CLI.
Minimal components
/status
)/subfiles/:id
)free_query_auth_token
for serviceNext steps
Client
Clearly state the limitations of our approach that while payment is minimized to a chunk at a time, We require 1 of n (provider) trust as there is no guarantee for a subfile to be completely torrented if no indexer serves the target files.
Minimal components
Assume the client is capable of identifying the correct ipfs hash and maintain budget balance on-chain
Next steps
Testing and Documentation
Goals of MVP
Scope outside of MVP
Post-MVP Developments
Once the MVP is successfully developed, tested, and validated, subsequent iterations would focus on