search

vincentsarago / gdal-http2-benchmark

GDAL 2.4 and HTTP2 benchmark
BSD 2-Clause "Simplified" License
5 stars 0 forks source link

S3 & HTTP/2 #1

Open pedros007 opened 5 years ago

pedros007 commented 5 years ago

Hi @vincentsarago ! I am also experimenting with the gdal-2.3 vsicurl HTTP options. I stumbled on to this repo while searching for a work-around to linker errors when building GDAL on Debian with a custom built version of curl with nghttp2 support. Your repo gave me the idea to try the GDAL configure flag --without-libtool which fixed my problem.

Anyway... I noticed you're accessing a file in S3. As far as I can tell, S3 does not support HTTP/2. When I run with CPL_CURL_VERBOSE=ON, it looks like all requests are HTTP/1.1. It seems like the order of magnitude improvement in number of CURL requests is thanks to GDAL_HTTP_MULTIRANGE and GDAL_HTTP_MERGE_CONSECUTIVE_RANGES.

Google Cloud Storage supports HTTP/2. I tried your tests with this file https://storage.googleapis.com/gcp-public-data-landsat/LC08/PRE/044/034/LC80440342016259LGN00/LC80440342016259LGN00_B11.TIF

using GDAL_HTTP_VERSION=1 and GDAL_HTTP_VERSION=2 and only saw a minor improvement in wall clock time.

vincentsarago commented 5 years ago

Hi @pedros007 👋

it looks like all requests are HTTP/1.1. It seems like the order of magnitude improvement in number of CURL requests is thanks to GDAL_HTTP_MULTIRANGE and GDAL_HTTP_MERGE_CONSECUTIVE_RANGES.

Yes you are right at the time, I miss understood what was going on here, the only gain comes from how GDAL is handling requests and not from http2 features 👌