Status code was not 200. Returned status code:401 ERROR AND SUGGESTION FOR FIX

[philipwlewis]

Hi, From this afternoon I was getting a 401 error code on the requests. I looked into the URLs and compared to PyTrends package which was still working and noticed the userConfig userType set to USER_TYPE_UNSPECIFIED

I added this into the payload creation and this seemed to fix the error

payload2$userConfig$userType <- "USER_TYPE_UNSPECIFIED"

Thanks for writing the nice package - hope this will help!

[eddelbuettel]

Still works here without it:

> library(gtrendsR)
> res <- gtrends("covid19")
> str(res)
List of 7
 $ interest_over_time :'data.frame':    260 obs. of  7 variables:
  ..$ date    : POSIXct[1:260], format: "2017-05-28" "2017-06-04" "2017-06-11" "2017-06-18" ...
  ..$ hits    : int [1:260] 0 0 0 0 0 0 0 0 0 0 ...
  ..$ keyword : chr [1:260] "covid19" "covid19" "covid19" "covid19" ...
  ..$ geo     : chr [1:260] "world" "world" "world" "world" ...
  ..$ time    : chr [1:260] "today+5-y" "today+5-y" "today+5-y" "today+5-y" ...
  ..$ gprop   : chr [1:260] "web" "web" "web" "web" ...
  ..$ category: int [1:260] 0 0 0 0 0 0 0 0 0 0 ...
 $ interest_by_country:'data.frame':    250 obs. of  5 variables:
  ..$ location: chr [1:250] "Kiribati" "Uganda" "Solomon Islands" "Togo" ...
  ..$ hits    : chr [1:250] "" "100" "" "" ...
  ..$ keyword : chr [1:250] "covid19" "covid19" "covid19" "covid19" ...
  ..$ geo     : chr [1:250] "world" "world" "world" "world" ...
  ..$ gprop   : chr [1:250] "web" "web" "web" "web" ...
 $ interest_by_region : NULL
 $ interest_by_dma    :'data.frame':    306 obs. of  5 variables:
  ..$ location: chr [1:306] "Alpena MI" "Honolulu HI" "Juneau AK" "Boston MA-Manchester NH" ...
  ..$ hits    : int [1:306] 100 60 58 55 54 54 53 53 53 52 ...
  ..$ keyword : chr [1:306] "covid19" "covid19" "covid19" "covid19" ...
  ..$ geo     : chr [1:306] "world" "world" "world" "world" ...
  ..$ gprop   : chr [1:306] "web" "web" "web" "web" ...
 $ interest_by_city   :'data.frame':    200 obs. of  5 variables:
  ..$ location: chr [1:200] "Kampala" "Bankura" "Agartala" "Bhiwani" ...
  ..$ hits    : int [1:200] 100 NA NA NA NA NA NA NA NA NA ...
  ..$ keyword : chr [1:200] "covid19" "covid19" "covid19" "covid19" ...
  ..$ geo     : chr [1:200] "world" "world" "world" "world" ...
  ..$ gprop   : chr [1:200] "web" "web" "web" "web" ...
 $ related_topics     :'data.frame':    20 obs. of  5 variables:
  ..$ subject       : chr [1:20] "100" "12" "6" "5" ...
  ..$ related_topics: chr [1:20] "top" "top" "top" "top" ...
  ..$ value         : chr [1:20] "Coronavirus disease 2019" "India" "Vaccine" "COVID-19 vaccine" ...
  ..$ keyword       : chr [1:20] "covid19" "covid19" "covid19" "covid19" ...
  ..$ category      : int [1:20] 0 0 0 0 0 0 0 0 0 0 ...
  ..- attr(*, "reshapeLong")=List of 4
  .. ..$ varying:List of 1
  .. .. ..$ value: chr "top"
  .. .. ..- attr(*, "v.names")= chr "value"
  .. .. ..- attr(*, "times")= chr "top"
  .. ..$ v.names: chr "value"
  .. ..$ idvar  : chr "id"
  .. ..$ timevar: chr "related_topics"
 $ related_queries    :'data.frame':    50 obs. of  5 variables:
  ..$ subject        : chr [1:50] "100" "71" "27" "22" ...
  ..$ related_queries: chr [1:50] "top" "top" "top" "top" ...
  ..$ value          : chr [1:50] "covid" "covid 19" "covid19 cases" "covid19 india.org" ...
  ..$ keyword        : chr [1:50] "covid19" "covid19" "covid19" "covid19" ...
  ..$ category       : int [1:50] 0 0 0 0 0 0 0 0 0 0 ...
  ..- attr(*, "reshapeLong")=List of 4
  .. ..$ varying:List of 1
  .. .. ..$ value: chr "top"
  .. .. ..- attr(*, "v.names")= chr "value"
  .. .. ..- attr(*, "times")= chr "top"
  .. ..$ v.names: chr "value"
  .. ..$ idvar  : chr "id"
  .. ..$ timevar: chr "related_queries"
 - attr(*, "class")= chr [1:2] "gtrends" "list"
> 

and

> plot(res)
> 

Could it be something at your end?

[PMassicotte]

I am also getting this error. Where comes from USER_TYPE_UNSPECIFIED?

[philipwlewis]

I looked at the url requests being made from the pytrends package that was still working for me https://pypi.org/project/pytrends/ and comparing the requests saw that it had the user type included unlike the gtrendsR request. I can't see where that is specified in the source code, but possibly it is picking up something that google sends back at some point? I am more of a hacker than being familiar with the API etc its odd that it worked for eddelbuettel (I teach a class and make use of the package for an assignment and a few students reported the issue). Let me know if I need to do anything on GitHub regarding the issue opening - this is my first time raising anything, Phil

[eddelbuettel]

We could triage, if we wanted to, over curl versions. It's a little strange that it bites in some cases but not in others. And I don't think the quick suggested fix -- already committed too !! -- will do any harm,

[PMassicotte]

It's a little strange that it bites in some cases but not in others

And I can not see what could cause this. I will try to investigate more.

@philipwlewis I fixed the issue in a separate branch. Maybe you and your students can give it a try.

[schoulten]

I was also getting this error so I tested the new branch version and the error went away.

[eddelbuettel]

Seems like the backend now tests for it. Odd.

Anyway, kudos to the pytrends team for the hint, to @philipwlewis for alerting us, and to @PMassicotte for a prompt update. Might be worth a routine quick update at CRAN too...

[PMassicotte]

I will prepare a submission today.