search

cavaliergopher / grab

A download manager package for Go
BSD 3-Clause "New" or "Revised" License
1.39k stars 150 forks source link

Make buffer size configurable #8

Closed andrebower closed 8 years ago

andrebower commented 8 years ago

Hey im downloading to an external hard drive and with the default buffer of 4096 bytes I'm getting really poor performance <1M/s. I manually increased the buffer size to 4096*1024 bytes and now im getting 6.2M/s (which is the maximum my internet connection offers).

cavaliercoder commented 8 years ago

You can now configure the transfer buffer size via Request.BufferSize. Please let me know if this is suitable.

I'm curious to know if the performance bottleneck is on reading from the socket or writing to the external disk. Can you try writing to a different device?

cavaliercoder commented 8 years ago

I haven't been able to replicate any change in performance increasing the buffer above 4kb. This buffer size was chosen to maximize counter updates without fragmenting disk writes which are typically in 4kb blocks. If the download speeds you are seeing are returned by Response.AverageBytesPerSecond, please consider that this is average over the life of the transfer (i.e., total bytes / time) and will not indicate speed at a point in time.

cavaliercoder commented 8 years ago

What OS and architecture are you using?

andrebower commented 8 years ago

Linux Arm banana pi 1. Average is what i want to max out and with 4kb it is really slow on my bananapi. I will test it again on my pc (osx and ssd) the next days.

andrebower commented 8 years ago

Hey i tested the new feature, works perfectly thank you! On my dev machine with a buffer of 4096byte my download speed is around 4Mbyte/s of 6Mbyte/s max. I doubt that it's somehow limited to my machine (i7-4770, SSD OCZ-TRION100, Mac OS X 10.11.5). Buffer of 4Mbyte solves that for me. On my target machine which is the banana pi I get 1Mbyte/s with 4k buffer and 6Mbyte/s with 4M buffer. Don't know if its because the external hdd or the bpi performance.