search

dgtlmoon / changedetection.io

The best and simplest free open source web page change detection, website watcher, restock monitor and notification service. Restock Monitor, change detection. Designed for simplicity - Simply monitor which websites had a text change for free. Free Open source web page change detection, Website defacement monitoring, Price change notification
https://changedetection.io
Apache License 2.0
17.09k stars 957 forks source link

Default `User-Agent` header could cause uninteded consequences #2407

Open Hritik14 opened 3 months ago

Hritik14 commented 3 months ago

This is more of an informative message than a bug. The default user agent is configured in /settings#fetching and is configurable. Although, some sites can behave differently when a browser user agent is supplied.

; curl 'https://jira.atlassian.com/rest/issueNav/1/issueTable' -H 'X-Atlassian-Token: no-check' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36' -w '%{http_code}\n'
XSRF check failed404

; curl 'https://jira.atlassian.com/rest/issueNav/1/issueTable' -H 'X-Atlassian-Token: no-check' -X POST -H 'User-Agent: curl' -w '%{http_code}\n'
400

Found this the hard way.

dgtlmoon commented 3 months ago

hey! yeah interesting, i was wondering if its better to just use the default 'curl' header agent (dont set it) and let people set their own i think

what you think?

Hritik14 commented 3 months ago

I think that would be the ideal choice. Also, it would be neat to show global headers as prefilled in the per-request headers so that it's more verbose