PBS spawns unlimited amount of goroutine

[linux019]

On high RPS to /openrtb2/auction endpoint PBS spawns more and more goroutines. To handle the huge amount of traffic I had to put an RPS limit on the load balancer. During normal work of PBS amount of goroutines is ~5K

PBS spaws a goroutine:

• to make call to bidder adapter (each bid request)
• call module hooks I propose to limit the amount of currently working goroutines and switch to goroutines pools. If we run out of workers PBS will return HTTP 503 instead of taking more and more traffic. On a high amount of goroutines golang scheduler spends a lot of CPU time to pick up the next goroutine. I can do this in fork but it’s a significant change to PBS core and will cause many merge conflicts

[bretg]

@linux019 - what library for goroutines pools are you proposing?

@zhongshixi has offered to provide a pointer to a potential solution.

@bsardo will coordinate a decision.

[SyntaxNode]

I had to put an RPS limit on the load balancer.

IMHO its good practice to use backpressure limiting layers in front of Prebid Server. This is the approach we use to avoid the situation described here.

I have no issue adding a goroutine limiting feature to PBS provided it doesn't add latency when unused due to either disabled (if we want to provide that option) or using a high limit value.

[Slind14]

I think this is more about reducing the compute spent on mcall at normal usage.

Being able to deal better with traffic spikes would be a side effect.

There should be no need to add a library for this.