Open grant proposal: `Sealing-as-a-Service`: `venus-cluster`

[Fatman13]

Open Grant Proposal: Sealing as a Service: venus-cluster

Name of Project: Sealing as a Service: venus-cluster

Proposal Category: core-dev

Proposer: Venus Team

(Optional) Technical Sponsor: n/a

Do you agree to open source all work you do on behalf of this RFP and dual-license under MIT, APACHE2, or GPL licenses?: Yes

Project Description

Sealing as a service has grown from an idea summarized in a Google doc to one of the priorities among Filecoin data-onboarding roadmap. We think that there are many factors contribute to the prioritization of the SaaS initiative.

High Entry Barrier

In order for a SP to grow its storage power, not only it has to make heavy investments on acquiring hardwares that are capable of computing PoRep proofs within reasonable time, but also it has to make itself fairly acquainted with operating, troubleshooting and monitoring its sealing pipeline. These two combined would turn away many potential smaller SPs, and many from other storage projects such as Chia.

Unbalanced Resources Allocation

With current storage providing model, each SP has to have its own sealing resources. However sealing demand for each SP may not be static as SP could be bounded by different factors to stop sealing and start later. This results in idling of sealing resources which could have been rented out to other SPs in need.

A SaaS model utilizing venus-cluster as its platform, where sealing resources can be pooled and used on-demand by SPs, could ultimately address both of the above problems. Additionally, an open SaaS platform like venus-cluster proposed in this open grant application could enable the emergence of SaaS providers who can then potentially leverage Venus platform to build their SaaS business.

Value

Merits of a SaaS solution are numerous, which may include lowering entry point to Filecoin network, more efficient usage of network's sealing resources, and the possibility of creating new line of business. Therefore, an open platform solution like venus-cluster is crucial for a SaaS provider to service their user base.

Proposed SaaS solution (venus-cluster) will be an all-purpose and generic solution which would exhibit the following traits...

• SaaS which can serve different miner_id at the same time
• SaaS which can be scaled quickly to meet SP's sealing needs
• SaaS which allows SP to retain control of their funds needed for PoRep
• A smart/customisable transfer scheme to send sealed sectors to SP (out of scope of this proposal)
• Sectors meta information for SP to continue PoST on sealed sectors

Risks may include trouble finding the initial cohort of providers and users to test the solution due to the obvious bottleneck of transferring sealed sectors from providers to users (SPs).

Deliverables

Three steps (architecture) of achieving SaaS with venus-cluster could be found here in details.

Development Roadmap

Milestone 1 - Improved Architecture & experiences

This milestone will be iterating on venus-clsuter's innovative architecture to lay the foundation for it to get ready for SaaS architecture 2 in the deliverables while improving sealing experiences in general.

For list of configurations, naming conventions and its effects, please refer to these two core documents (1 and 2) to learn more.

Milestone 1.1

Deliverables:

• Improve sealing experiences and job scheduling
  • Add weights to processors of same sealing phase
  • Add rate limiter when allocating multiple PC1 tasks
  • Improve abort command on both venus-sector-manager and venus-worker
  • Improve scheduling of wdPost
  • Improve venus-worker log output formats
• Architectural improvement
  • Allow sealing_thread to wait for deals according to configurations
  • Add new sealer plan
  • venus-worker improved cuda support
• Corresponding documentations to all aforementioned changes

Funding for milestone 1.1:

$64,640

Estimated Milestone 1.1 Delivery:

4 weeks

Milestone 1.2

Deliverables:

• Improve sealing experiences and job scheduling
  • Improve sector related CLI (multi miner_id support)
  • Improve retry logic
  • Improve Snapup abort process (updated states)
  • Add detection toolchains for fine tuning sealing pipeline
  • Improve configuration UX
• Architectural improvement
  • Improve PC1 memory affinity with configured numa node
  • Add external processor support for venus-worker to allow winningPost
  • Improve sector migration support (free flow of Lotus sector to Venus and vice versa)
  • Add warmup logic for PoST
• Corresponding documentations to all aforementioned changes

Funding for milestone 1.2:

$64,640

Estimated Milestone 1.2 Delivery:

4 weeks

Milestone 2 - SaaS PoC

This milestone will be realizing the SaaS architecture 2 proposed in the deliverables.

Again, for list of configurations, naming conventions and its effects, please refer to these two core documents (1 and 2) to learn more.

Deliverables:

• Improve sealing experiences and job scheduling
  • Add weights to processors of same sealing phase
  • CLI improvements
  • Support for loading local piece files
  • Improve AP memory efficiency and add rate limiter
  • Improve message sending logic
  • Improve cgroup configuration and affinity to hardware
• Architectural improvement
  • Add sector rebuild feature which allows user to repair sector due to hardware failures
  • External processor for custom transfer strategy and corresponding configurations
  • External processor for custom AP strategy and corresponding configurations
• Corresponding documentations to all aforementioned changes

Funding for milestone 2:

$64,640

Estimated Milestone 2 Delivery:

4 weeks

Milestone 3 - SaaS

This milestone will be realizing the SaaS architecture 3 proposed in the deliverables.

More deliverables maybe added to existing listing as the development work carries out.

Deliverables:

• Improve sealing experiences and job scheduling
  • Metrics integration for usability
• Architectural improvement
  • External processor wdPost support for parallel subtasks
  • Specialized venus-sector-manager for SaaS provider
  • Configuration manager with hot reload support
• Corresponding documentations to all aforementioned changes

Funding for milestone 3:

$64,640

Estimated Milestone 3 Delivery:

4 weeks

Milestone 4 - Compatibility with Lotus and Boost (Optional)

This milestone will have venus-cluster be compatible with lotus/boost, which enables venus-cluster to work with lotus daemon and boost effectively allowing it to be used interchangeably with lotus-miner.

Deliverables:

• Compatibility with Boost API
• Compatibility with Lotus API

Funding for milestone 4:

$129,280

Estimated Milestone 4 Delivery:

8 weeks

Total Budget Requested

Total budget = $258,560 + $129,280 (optional) Weekly budget ~= $16,160 with a team of Senior Architect 1, Senior Backend Developer 2, Quality Assurance Engineer 1, Project Manager 1

Maintenance and Upgrade Plans

The goal of the team is to support venus-cluster long term, which including continuously adding more features that community deemed worthy. Therefore, further perfecting venus-cluster as the platform for SaaS type of solution.

Team

Team Members

Force community engineering team

Team Member LinkedIn Profiles

Team Website

https://forcecommunity.io/

Relevant Experience

Force community has been an active contributor to Web3 ecosystem and Filecoin ecosystem in general. The engineering team from Force community has a track record of contributing code to Lotus as far back as Testnet and Space Race.

Team code repositories

https://github.com/ipfs-force-community

Additional information

Force community is committed to become a major contributor to Web3 infrastructure and we see Filecoin at the core of the big Web3 migration. We hope that we could fast track the realization of Web3 adoption by contributing our software development capacity to the course and join hand in hand with all other ecosystem developers around the globe through this historical journey!

[realChainLife]

@jennijuju when you have a minute can you review this proposal and share your thoughts. Is this a solution that can be useful on Lotus as well, precisely as an alternative to the sealing manager process planned for development.

[jennijuju]

@Fatman13 can you confirm wether this proposal is to:

• enable SaaS API interface in venus cluster
• or make a SaaS a builtin feature/service within Venus-cluster, like the Venus node service
• or both?

[Fatman13]

@Fatman13 can you confirm wether this proposal is to:

enable SaaS API interface in venus cluster or make a SaaS a builtin feature/service within Venus-cluster, like the Venus node service or both?

Hello, @jennijuju! Thanks for asking! This proposal enables any potential SaaS provider to use venus-cluster as its platform for its SaaS service to SPs. So yes to both. The team is more than happy to have a call if you have any further questions!

[Fatman13]

@realChainLife Hello, team! Wondering if there is any progress on this application yet?

[ErinOCon]

Hi @Fatman13, I know it has been a considerable amount of time since we have provided an update for this proposal. In light of the macroeconomic climate, the review of our budget and priorities for the fiscal year has resulted in a longer evaluation period. Thank you for all of your patience.

Unfortunately, we will not be moving forward with a grant at this time. We are wishing you all the best as you continue to build!

[Fatman13]

In light of the macroeconomic climate, the review of our budget and priorities for the fiscal year

Understandable. 🙏