Me: Is it possible to make the tier1 "ramp up" requests to the tier2 (1 new thread every 0.25 seconds or something). When I start with substreams-tier1-max-subrequests: 1000 and make a new connection, it seems like it overloads the s3 cluster. I need see if I can isolate the actual problem.
Alex: yeah, maybe not a bad idea.. maybe the scheduler can pace things out a bit,
0.25 seconds would be okay to solve your issue? Probably wouldn’t affect the end-user perception that much
Me: I'm not sure 0.25 is enough or too much... maybe make it configurable? (0.25 seconds on 1000 threads is 4 minute delay)
matthewdarwin
commented
1 year ago
Me: Is it possible to make the tier1 "ramp up" requests to the tier2 (1 new thread every 0.25 seconds or something). When I start with substreams-tier1-max-subrequests: 1000 and make a new connection, it seems like it overloads the s3 cluster. I need see if I can isolate the actual problem.
Alex: yeah, maybe not a bad idea.. maybe the scheduler can pace things out a bit, 0.25 seconds would be okay to solve your issue? Probably wouldn’t affect the end-user perception that much
Me: I'm not sure 0.25 is enough or too much... maybe make it configurable? (0.25 seconds on 1000 threads is 4 minute delay)