search

valyala / fasthttp

Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http
MIT License
21.96k stars 1.76k forks source link

Sending ~1M GET requests with PipelineClient #831

Open jakobhuss opened 4 years ago

jakobhuss commented 4 years ago

Hello,

I just stumbled across this library and it seems to be very powerful. I hope it is okay to ask questions here, otherwise please let me know where I should post them. I need some pointers on how to use fasthttp and more specifically the PipelineClient.

First of all not all servers support pipelining requests. How can I probe a server if pipelining works? Should I just try the PipelineClient and see if there are a lot of errors and then use the other clients as fall-back?

Second question is about load balancing across multiple hosts. What would be the way to from a host name send a dns query retrieve all A and AAAA records. And then create connections to all of them using the PipelineClient?

Thirdly how could I optimize paramters during runtime like MaxConnections and MaxPendingRequests?

erikdubbelboer commented 4 years ago

I guess you could test if a server supports pipelining by sending a batch of multiple requests and measure the time between responses to see if they arrive in the same batch of with a delay between them. In theory I don't think PipelineClient should give any errors for servers that don't support it, it will just be slow.

With multiple hosts you mean different hostnames or multiple hosts on the same hostname?

Those are parameters you'll probably have to just try to find a good value that works for you. This completely depends on how much you client and the server can handle.

May I ask what kind of requests you are going to do that requires so many?

jakobhuss commented 4 years ago

Thanks for you reply. I will check out the delay between responses. That could be a way of doing it.

With multiple hosts I mean multiple hosts on the same hostname. Spreading the load across several IP:s. I guess this could be done with LBClient and feeding a few PipelinceClients to it.

I understand that the parameters are very dependant on the client/server situation. I was thinking of how to dynamically adapt this for the situation during runtime.

Yes you may. I'm playing with a http fuzzer for finding misconfigurations in http servers. It would be good for black box testing during penetration testing or bug bounty. There are big lists of files that should not be reachable for instance .htpasswd. And I will look for files like that. I think this library and the PipelineClient could help me increase the speed of this task by a lot.

erikdubbelboer commented 4 years ago

Indeed just usenet.LookupHost and create a PipelinceClient for address and add them to a LBClient.

MaxConnections and MaxPendingRequests can't be changed after the PipelinceClient is in use so you can't dynamically adapt.

To be honest, I don't think PipelinceClient is going to make much of a difference there. PipelinceClient is only really useful if the network latency is a bottleneck. This sounds more like the bottleneck is going to be the webserver and it's filesystem.

jakobhuss commented 4 years ago

Thanks again, I tried exactly what you suggest with LBClient. I will have to look at some throtteling, perhaps by creating new PipelineClients, to ensure I'm not choking the servers or step back when status 429 or 5xx is returned.

Regarding the PipelineClients performance I have not compared to the other clients in fasthttp. But the initial performance of rps in my tests looks very promesing. Threre are a few blog posts out there describing how pipelining can speed up requests for this usecase. This tool, Turbo Intruder, is using pipelineing for instance.

ip-rw commented 4 years ago

Pipelining is terrific... but support is hit and miss at best. https://github.com/valyala/httpteleport worth a look.

jakobhuss commented 4 years ago

Pipelining is terrific... but support is hit and miss at best. https://github.com/valyala/httpteleport worth a look.

That is a very cool idea. However it won't work when you only can act as a client, right?

I think my plans to use pipelining (when supported) and tune parameters will work best in this case. I have yet not a great idea for testing server support for piplining with fasthttp.

erikdubbelboer commented 4 years ago

If you only have control over the client httpteleport is not going to work.