Flag for disabling http/2

storj / edge

Storj edge services (including multi-tenant, S3-compatible server to interact with the Storj network)

GNU Affero General Public License v3.0

53 stars 18 forks source link

Flag for disabling http/2 #430

Closed NiaStorj closed 4 months ago

NiaStorj commented 5 months ago

Summary

Context

We have observed that the performance of backups using Veeam on Storj seems to degrade, which might be linked to the HTTP/2 protocol's handling of our data transfer.

Expected Outcome from Disabling HTTP/2

Enhanced Transfer Speed: Without the multiple stream capabilities of HTTP/2, we expect to avoid prioritization and resource allocation issues leading to higher throughput
Ruling out HTTP/2: If there are no noticable changes, we can at least rule out our gateway's use of HTTP/2 as a contributing factor

Acceptance Criteria

Disable HTTP/2 on the gateway
@derek-storj to run benchmarks against gateway with HTTP/2 disabled

derek-storj commented 5 months ago

@wthorp and @derek-storj to explore HTTP proxy options on Windows Server and report back Veeam testing results.

derek-storj commented 4 months ago

Placing a proxy in front of Veeam has proven to be a bottleneck, so did some testing with rclone with a workload that mimics Veeam behavior:

Created a folder with 20GB worth of 4MB files, then used rclone to copy that folder to Storj.

64 concurrent transfers, HTTP2 enabled -> 2:37 (131MB/s) 64 concurrent transfers, HTTP2 disabled -> 0:57 (380MB/s), ~3x improvement

(Running netstat from Windows CLI confirms HTTP2 disabled shows 64 connections opened to 136.0.77.2, versus HTTP2 enabled shows 1 connection)

derek-storj commented 4 months ago

https://storj.slack.com/archives/C06BDR8SC5Q/p1715186028011289

storj-gerrit[bot] commented 4 months ago

Change pkg/httpserver: support disabling HTTP/2 for gateway-mt mentions this issue.