sparklemotion / mechanize

Mechanize is a ruby library that makes automated web interaction easy.
https://www.rubydoc.info/gems/mechanize/
MIT License
4.39k stars 473 forks source link

tidy: do not send deprecated headers #647

Closed flavorjones closed 1 month ago

flavorjones commented 1 month ago

See discussion at #646 which indicates that sending the now-deprecated accept-charset header confuses some (presumably buggy, but real-world) servers.

I'm opening this ticket to drive:

Special thank-you to @MarcoPrins and @tomasz-jasiulek for reporting and investigating this.

flavorjones commented 1 month ago

Inventory

A vanilla request sends the following headers:

accept-charset

See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Charset

Warnings: "Do not use this header. Browsers omit this header and servers should ignore it."

accept-encoding

See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding

Warnings: none

accept-language

See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Language

Warnings: none

accept

See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept

Warnings: none

user-agent

See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent

Warnings: only to servers about browser detection using the user agent being a bad idea

host

See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Host

Warnings: none

flavorjones commented 1 month ago

Seems like accept-charset is the only offender. See #648.