User agent header seems to be not sent

[dpaulino]

According to reports

[dpaulino]

Can't seem to repro. See nightingale on the right supply a user-agent, and see aspnet core 5 on the left recognize that user-agent

[dpaulino]

Closing as no repro

[jtippet]

The problem seems to involve the characters ;,<=>\?@[]{} (unless they are contained in parentheses) in the user-agent.

test works ok:

te;st is silently ignored:

t (e;s) t works ok:

I think you can argue that these UAs are technically illegal (my reading of RFC 7231/7230 says that a UA must not contain any of (),/:;<=>?@[\]{} , except / is allowed once before a product-version token). But since Nightingale is a tool specifically for developers, I claim that Nightingale should ideally allow devs to break this rule, if they specifically ask to. By comparison, curl -A "a;b;;(),/:;<=>?@[\]{}" https://whatmyuseragent.com/ dutifully sends that tacky user agent to the server.

This isn't a completely theoretical problem -- here are some real-world UAs that Nightingale cannot send:

• The duckduckgo search engine's crawler has a stray semicolon: DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
• This Google WAP proxy had adjacent parentheticals: DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
• The AppleTV uses some commas in its UAs: AppleTV/7.5 iOS/8.4.4 AppleTV/7.5 model/AppleTV3,2 build/12H885 (3; dt:12) or com.nfl.gamecenter/1.0.1 ios/13.3 appletv/13.3 model/appletv5\,3 hwp/t7000 build/17k449 (3; dt:119).

I'm not sure how Nightingale's HTTP client is implemented, but this suspicious item turns up in a web search.

[MahdiGhiasi]

I'm seeing the same behavior @jtippet explained.

Specifically, I was trying to repliacte Google's FeedFetcher bot request which contains a semicolon that is not contained in parantheses.

It is FeedFetcher-Google; (+http://www.google.com/feedfetcher.html), and Nightingale refuses to send this user agent.