Goal: optimise costs, reduce waste; better experience on chains with less traffic.
Problem: when launching multiple chains, you need to launch a tier1 and a tier2 stack for all; sometimes tier2 are dormant, so a HorizontalPodAutoscaler will scale down the deployment; yet when someone arrives with 200 jobs, it overloads and starts crashing.
Solution: create a single pool, large pool, of tier2 nodes.. that will receive their configuration from the tier1 upon request. This would also make it less drastic when upgrading tier2 nodes, because there would be a more fluid rotation, having more tier2 nodes to cycle through. Large power would be immediately available to someone bursting with requests, as everything would be shared.. and we could then scale up more gradually with the HPA when they start being saturated.
Prerequisites:
the substreams tier2 would need to have all extensions, and should enable extensions only when requested.
we could pick up the eth_call export from the provided WASM to enable an extension, and implement the explicit extensions support like in #340
today, the only extensions is eth_call, so a stock firehose-ethereum would be able to serve all requests.
Proposal:
take out all the flags from the tier2 node's CLI,
tweak the gRPC request between tier1 and tier2; pass all flags from the tier1 to the tier2 that depend on the network:
--common-metering-plugin=grpc://localhost:9010?buffer=100000&network=arb-one <-- network name should be passed, and given to the metering calls
--common-merged-blocks-store-url=gs://path-to-bucket/arb-one/v1?project=project_id
--common-first-streamable-block=0
--substreams-state-bundle-size=1000
--substreams-state-store-url=gs://path-to-another-bucket/arb-one/substreams-states?project=project_id
--substreams-state-store-default-tag=v1 <--- the selected tag should be computed and passed
--substreams-rpc-cache-store-url= <-- unused now, going away
--substreams-rpc-endpoints=http://rpc-caching-proxy:8080/ <--- this is only for firehose-ethereum
--substreams-rpc-gas-limit=0 <-- unused with wazero
rework the tier2 code so that those things are initialized upon request, and not on boot.
do we need a pool of clients per gs://path-to-bucket ? or we can just create a new client on each request?
deployment wise, the rpc-caching-proxy currently is network specific, so the URL could be passed by the tier1, and we could have a network-specific rpc-caching layer.
matthewdarwin
commented
5 months ago
Goal: optimise costs, reduce waste; better experience on chains with less traffic.
Problem: when launching multiple chains, you need to launch a tier1 and a tier2 stack for all; sometimes tier2 are dormant, so a HorizontalPodAutoscaler will scale down the deployment; yet when someone arrives with 200 jobs, it overloads and starts crashing.
Solution: create a single pool, large pool, of tier2 nodes.. that will receive their configuration from the tier1 upon request. This would also make it less drastic when upgrading tier2 nodes, because there would be a more fluid rotation, having more tier2 nodes to cycle through. Large power would be immediately available to someone bursting with requests, as everything would be shared.. and we could then scale up more gradually with the HPA when they start being saturated.
Prerequisites:
eth_callexport from the provided WASM to enable an extension, and implement the explicit extensions support like in #340eth_call, so a stockfirehose-ethereumwould be able to serve all requests.Proposal:
tier2node's CLI,gs://path-to-bucket? or we can just create a new client on each request?rpc-caching-proxycurrently is network specific, so the URL could be passed by the tier1, and we could have a network-specific rpc-caching layer.