search

cs531-f19 / discussions

Discussions board for CS 431/531 Web Server Design course
2 stars 12 forks source link

Access denied for some user agents in the wild #57

Open ibnesayeed opened 4 years ago

ibnesayeed commented 4 years ago

Find an example "in the wild" where the access to the resource is denied by default when using cURL, but not when specifying a different user agent (e.g., -A "Mozilla").

felixvelariusbos commented 4 years ago

https://reddit.com

curl -I https://www.reddit.com/ returns a 401 error curl -IA "Chrome" https://www.reddit.com/ returns 200 + content

>> curl -I https://www.reddit.com/
HTTP/2 401
cache-control: private, s-maxage=0, max-age=0, must-revalidate
set-cookie: session_tracker=ipBM0xjzBj34yVQJBa.0.1571792334000.Z0FBQUFBQmRyNlhPT3V5cXZ3VG9QNjg2YnVTVjliNjk5cU8yakgxXy03b1plV2VWNi1rTG9PNlQxM3VDWm1TMzdjTnBEc0doS3dpUHFGVDRWVTkybWxmdmRYZVdpNW5wREh1dFBORFpmWlk4RkFvczVKUVJ6R3AxNWRuZTNRekRNSFpEX0FfNWN3MTM; path=/; domain=.reddit.com; secure
x-frame-options: SAMEORIGIN
accept-ranges: bytes
date: Wed, 23 Oct 2019 00:58:54 GMT
via: 1.1 varnish
x-served-by: cache-dca17759-DCA
x-cache: MISS
x-cache-hits: 0
x-timer: S1571792334.165764,VS0,VE196
vary: Accept-Encoding
set-cookie: edgebucket=qcEOvhIRW652z5Cd1F; Domain=reddit.com; Max-Age=63071999; Path=/;  secure
strict-transport-security: max-age=15552000; includeSubDomains; preload
server: snooserv
>> curl -IA "Chrome" https://www.reddit.com/
HTTP/2 200
cache-control: private, s-maxage=0, max-age=0, must-revalidate
set-cookie: loid=00000000004utx8e2f.2.1571792536890.Z0FBQUFBQmRyNmFZRXByV0QtY0JXclZDalhVUkZteXdLdmVhb0U3NWtuak5pTzV6VmNlVkZGRDdmcmh4ZkZuTFdsaGRRUzF2Nk5XVkpULWtERklrb05JbzY0VXIyNGVQNGY2NFBLeW1rdGdMdzMyUU1UdXRsYnNyNnV5Tks5bE5ONGh0RTRONllpVU4; path=/; expires=Sat, 23 Oct 2021 01:02:16 GMT; domain=.reddit.com; secure
set-cookie: d2_token=3.38ff21f5c8e4d83a5c0cba0901ae9d9b2c4898e0f55382fc26b3a6301f4e362c.eyJhY2Nlc3NUb2tlbiI6Ii1pbjFtUk55dnJRSnN1OTdzVmR5MF84SXBDRUUiLCJleHBpcmVzIjoiMjAxOS0xMC0yM1QwMjowMjoxNi4wMDBaIiwibG9nZ2VkT3V0Ijp0cnVlLCJzY29wZXMiOlsiKiIsImVtYWlsIl19; path=/; expires=Sat, 23 Oct 2021 01:02:16 GMT; domain=.reddit.com; secure
set-cookie: session_tracker=q2oeUh7plqO3EFfWyN.0.1571792536000.Z0FBQUFBQmRyNmFZREZQLW1ETFZnV2JCdWg5UHhSMFlMZWtMcFFqSmtVdW1abnRRdjZRNXNyMkhqRDM1OHQ4NUs2RmtyNGFyMmNjVjBIM25aNU5oc1lmTlRKLTlxSWFNTVRIM2ZtXzd2N2lySThEV2tsTWNUZ09JbHhPUG53Zjg1Und5UXYxbnN0VHQ; path=/; domain=.reddit.com; secure
x-frame-options: SAMEORIGIN
accept-ranges: bytes
date: Wed, 23 Oct 2019 01:02:17 GMT
via: 1.1 varnish
x-served-by: cache-dca17748-DCA
x-cache: MISS
x-cache-hits: 0
x-timer: S1571792536.262584,VS0,VE1241
vary: Accept-Encoding
set-cookie: edgebucket=za3j2VUOMJVi1BTSmM; Domain=reddit.com; Max-Age=63071999; Path=/;  secure
strict-transport-security: max-age=15552000; includeSubDomains; preload
server: snooserv
ibnesayeed commented 4 years ago

@felixvelariusbos please post the output (of GET or HEAD using curl) as well, not just describe it. You can use three back-ticks before and after your output code to format it properly.

felixvelariusbos commented 4 years ago

@ibnesayeed Sorry about that, edited the comment to add the head output.

ibnesayeed commented 4 years ago

While it is not a 403 response, you have still discovered something interesting here so your example is accepted and will be graded accordingly.

RFC 7235 says:

The server generating a 401 response MUST send a WWW-Authenticate header field ...

but Reddit is failing to include one.

kritikagarg commented 4 years ago

$ curl -IL http://www.163.com HTTP/1.1 403 Forbidden Date: Mon, 11 Nov 2019 04:49:24 GMT Content-Type: text/html Content-Length: 2653 Connection: keep-alive Server: Cdn Cache Server V2.0 Expires: Mon, 11 Nov 2019 04:49:24 GMT X-Via: 1.0 PSjnddldTOR1lg48:6 (Cdn Cache Server V2.0) Cache-Control: no-cache,no-store,private

$ curl -ILA "chrome" http://www.163.com HTTP/1.1 200 OK Date: Mon, 11 Nov 2019 04:49:35 GMT Content-Type: text/html; charset=GBK Connection: keep-alive Expires: Mon, 11 Nov 2019 04:50:49 GMT Server: nginx Vary: Accept-Encoding Cache-Control: no-cache,no-store,private X-Via: 1.1 PSjsczsx2di47:0 (Cdn Cache Server V2.0), 1.1 hx173:8 (Cdn Cache Server V2.0), 1.1 PSjnddldTOR1lg48:6 (Cdn Cache Server V2.0)

himarshaj commented 4 years ago

C:\Users\Himarsha>cURL -IL www.arabiadate.com HTTP/1.1 403 Forbidden Server: nginx Date: Mon, 11 Nov 2019 07:45:26 GMT Content-Type: text/html Content-Length: 162 Connection: keep-alive

C:\Users\Himarsha>cURL -ILA "mozilla" www.arabiadate.com HTTP/1.1 200 OK Server: nginx Date: Mon, 11 Nov 2019 07:46:16 GMT Content-Type: text/html; charset=UTF-8 Connection: keep-alive Vary: Accept-Encoding X-Check: 3c12dc4d54f8e22d666785b733b0052100c53444 X-Language: english X-Template: tpl_CleanPeppermintBlack_twoclick X-Buckets: bucket084 X-Adblock-Key: MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALquDFETXRn0Hr05fUP7EJT77xYnPmRbpMy4vk8KYiHnkNpednjOANJcaXDXcKQJN0nXKZJL7TciJD8AoHXK158CAwEAAQ==_jFVIY+uc6VV+Az3a8/tJrnwW/xZta4qtGuDHv2bzARvRpEWjDNdntcIowUoqVcsfSvePXcPXoHEwBfKaBH/e5g==