search

daurnimator / lua-http

HTTP Library for Lua. Supports HTTP(S) 1.0, 1.1 and 2.0; client and server.
https://daurnimator.github.io/lua-http/
MIT License
777 stars 80 forks source link

Possible HTTP/2 Incompatibility #224

Open sparked435 opened 4 weeks ago

sparked435 commented 4 weeks ago

This might not be a lua-http issue; but only lua-http shows this behavior on sites using the Fastly CDN. The following works, outputting "200", as expected:

local req = httpreq.new_from_uri( 'https://www.bbc.com/robots.txt' ) req.version = 1

local headers, body = req:go(5) print(headers:get(':status'))

Switching req.version to either nil or 2 returns a 403. I spent some time ensuring Curl and lua-http were sending identical headers; lua-http gets 403 for HTTP/2 and Curl always gets a 200 regardless of version. Web browsers consistently succeed as well.

Environment is Lua 5.4, lua-http 0.4, cqueues 20200726.54, and luaossl 20220711.

daurnimator commented 2 weeks ago

It appears as though BBC have opted into some kind of fastly "bot-prevention" feature?

sparked435 commented 2 weeks ago

Whatever Fastly is doing was probably was intended to be something like that.

That said, I can run a few hundred requests with lua-http returning 403, and then run Curl with the exact same headers from the exact same IP and get a 200 - so it's upset at something lua-http specific and not the IP appearing to act like a bot/scraping/hitting an denied URL.

Is there a way to force lua-http to speak HTTP/2 but over unencrypted TCP? That way I could use stunnel and try to debug further with a packet sniffer.

daurnimator commented 2 weeks ago

Is there a way to force lua-http to speak HTTP/2 but over unencrypted TCP? That way I could use stunnel and try to debug further with a packet sniffer.

sure. you should be able to use a http url and then set

req.version = 2