chrisabruce
commented
2 years ago Is this proposal just to use your CDN as a front to the Helium Public API? Or is this intended to attempt to run full ETL/Postgres/API?
Open PUSHRCDN opened 2 years ago
chrisabruce
commented
2 years ago Is this proposal just to use your CDN as a front to the Helium Public API? Or is this intended to attempt to run full ETL/Postgres/API?
PUSHRCDN
commented
2 years ago Is this proposal just to use your CDN as a front to the Helium Public API? Or is this intended to attempt to run full ETL/Postgres/API?
Hello, we intend to run full ETL, database and API, and will be proxying it via our CDN.
chrisabruce
commented
2 years ago Who is the intended audience of this and what amount of traffic are you expecting to handle? Is this expected to replace Helium public API? How are the ongoing operational costs to be calculated and paid?
The public API does well over 3bn requests/month and growing fast (it was under 1bn req/month 6 months ago). It bursts up to 300,000 req/sec (not a typo). It is running on over 1200 CPU cores for just the databases (20 replicas) and growing. All bare metal 60/100 core Ryzens. It currently uses Fastly for CDN. The API servers are also running on a cluster of servers. And all this is even with 1 req/sec rate limiter in place.
I am not sure, but the load, traffic, and amount of management effort might be being way underestimated?
PUSHRCDN
commented
2 years ago I guess there might be some misunderstanding. This proposal is based on https://gist.github.com/dansku/17daa20314610ab0f8b3a73a8dec38b8 , which is linked to from https://heliumfoundation.notion.site/The-Helium-Foundation-Grant-Program-18a0484cf8a1494bae9eb3497b77a3cf , which appears to seek decentralization for the API and to incentivize more public API endpoints to be created. The purpose is not to replace Helium's endpoint, If I understand it correctly, but to add another endpoint which could be used publicly to help offloading Helium's API. Operational costs are not mentioned in the gist, for which I have assumed that is subject open to discussion based on another document here https://docs.google.com/document/d/1KOQchoCyqnZYDUfYxMa1B7UL-oVUIqHCLPdUd-fNRaw/edit , under the Work opportunities section, which lists the option to commercialize for heavy use as a requirement. Please correct me if I am wrong.
chrisabruce
commented
2 years ago Can you try and estimate the load and duration that this setup would support for the project amount?
Project:
Public Helium API
Elevator Pitch:
This submission aims to deliver a high-performance, scalable and highly available endpoint for the public Helium API.
Total fiat/hnt ask:
$50 000 as listed in the published contract work opportunity
Team website:
This service will be built, maintained and operated by Net Stack Ltd, the company behind PUSHR
Project Details:
This API service is built on top of pairs of bare-metal 16-core AMD Ryzen servers powered by NVMe drives. Each server in a pair is load balanced and has an proactive failover setup with realtime monitoring. All pairs are load balanced with proactive failover capability on per-pair basis. Scalability is simplified and there is no practical limit on the pairs of servers that can be added, as well as no limit on their geographical distribution. The API is proxied via a global network of edge servers with proactive failover in the event of edge failure. Custom Layer 7 attack mitigation and rate limiting rules are applied on the network edge. This service is designed to be commercialized for heavy use.
Roadmap: