Closed maelle closed 4 years ago
good question.
ok does set a user agent string by deafult, like:
ok("https://google.com", verbose=TRUE)
#> > HEAD / HTTP/1.1
#> Host: google.com
#> User-Agent: libcurl/7.54.0 r-curl/4.2 crul/0.9.0.9100
the user can choose to change the ua string like
ok("https://google.com", useragent = "hello world", verbose = TRUE)
#> > HEAD / HTTP/1.1
#> Host: google.com
#> User-Agent: hello world
in your eg url above, i guess we can't do anything automatically, but we can document this. for example, tell users that a FALSE
may be incorrect depending on their use case, e.g, if they want to know if curl based scraping will work without fiddling with curl options, then the FALSE
is probably correct, but if they want to fiddle with curl options, then first step would be to send verbose=TRUE
so they can see whats going on with any redirects and headers. And then talk about user agent strings and some websites blocking based on user agent strings.
Created on 2019-11-13 by the reprex package (v0.3.0)