search

levigross / grequests

A Go "clone" of the great and famous Requests library
Apache License 2.0
2.14k stars 137 forks source link

Grequests should not replace empty UserAgent with it's own #68

Closed ehmo closed 5 years ago

ehmo commented 6 years ago

As much as I like Grequests and I truly want more people to find about it, forcing default User-Agent is not the way.

https://github.com/levigross/grequests/blob/fd07961896a93d37e169271bb23ee423c42033d3/request.go#L558

Many platform will change behavior based on the User-Agent. Some of their WAF (web application firewall) will try to single out requests originating from the same IP, User-Agent, etc. There are situation where an empty User-Agent needs to be sent, literally User-Agent: and Grequests is preventing that from happening.

So I would suggest two solutions.

1) Create custom struct that can be set to empty, e.g.

// NullString represents a string that may be null.
type NullString struct {
    String string
    Valid  bool // Valid is true if String is not NULL
}

and if Valid is true then use the String provided in the struct.

2) Remove the check.

levigross commented 5 years ago

Hi Ehmo,

Thank you for using the library. I would reccomend setting the value to " " which is technically still blank. You can also use the OnRequest hook. I am planning on changing the behavior of the request setting functionality in the near future which will address this issue.

ehmo commented 5 years ago

Using " " is not a good solution, as it's visible on the server side. If you are using the library for scrapping, it will cause troubles.

I do believe my original comment stands. You should not force the library on the User-Agent. I understand why are you doing it, but people like it plenty without the need of advertising it over the HTTP and causing developers to jump through hoops to deal with it.