In a situation where there is lots of fast scraping activity mitmproxy can burn a giant percentage of the overall cpu of the host. Currently, it's pegging at about 300% cpu on an 8 processor machine. This is probably short-lived (as it's not something we see a huge amount of) yet it does point at that mitmproxy is probably not the best long-term option for the job it's doing.
It might be worth looking at reimplementing the proxy.
I'd take a stab at guessing that doing it with golang wouldn't require a huge amount of code and would be fast.
In a situation where there is lots of fast scraping activity mitmproxy can burn a giant percentage of the overall cpu of the host. Currently, it's pegging at about 300% cpu on an 8 processor machine. This is probably short-lived (as it's not something we see a huge amount of) yet it does point at that mitmproxy is probably not the best long-term option for the job it's doing.
It might be worth looking at reimplementing the proxy.
I'd take a stab at guessing that doing it with golang wouldn't require a huge amount of code and would be fast.