Server response: 429 & Error in `mutate()`

[Iannnn1]

Hi,

Great package!

When I use ts_gtrends_mwd, I encounter the issue of "Server response: 429 - too many requests," and completely interrupts the process due to an "Error in mutate()," for instance,

......
Downloading data for 2023-10-05 2024-01-05
Server response: 429 - too many requests
Waiting for 20 seconds
Retrying...
Attempt 2/10
Downloading data for 2023-10-06 2024-01-06
Downloading data for 2023-10-07 2024-01-07
Server response: 429 - too many requests
Waiting for 20 seconds
Retrying...
Attempt 2/10
Downloading data for 2023-10-08 2024-01-08
......
Downloading data for 2023-10-30 2024-04-04
High frequency series shorter than low frequency. Discarding low frequency before 2022-12-04.
High frequency series shorter than low frequency. Discarding low frequency before 2022-12-04.
Downloading data for 2022-11-30 2023-05-30
......
Downloading data for 2022-05-06 2024-04-04
Server response: 429 - too many requests
Waiting for 20 seconds
Retrying...
Attempt 2/10
Downloading data for 2022-05-13 2024-04-04
......
Downloading data for 2023-09-30 2024-04-04
Server response: 429 - too many requests
Waiting for 20 seconds
Retrying...
Attempt 2/10
Server response: 429 - too many requests
Waiting for 40 seconds
Retrying...
Attempt 3/10
Error in `mutate()`:
ℹ In argument: `trend_data = list(...)`.
ℹ In row 11.
Caused by error in `curl::curl_fetch_memory()`:
! Timeout was reached: [trends.google.com] Send failure: Connection was reset
Run `rlang::last_trace()` to see where the error occurred.

The situation I most often encounter is that an error occurs directly when starting. I have updated R and the packages, but it doesn't seem to have resolved the issue. I would appreciate your help.

I would also like to seek advice, as I've found that the data obtained through ts_gtrends_mwd() function can contain negative numbers and values > 100. Is it reasonable to directly convert these to the range of 0-100?

Thanks! @karoliskoncevicius

[karoliskoncevicius]

Hello and sorry for such a prolonged response. I received another report about the same issue, so I will repeat the answer here.

The issue is about server response 429 - too many requests. This usually means that Google servers think you are making too many requests and in the past you would expect to see this response when too many keywords are downloaded in succession. The work-around for that was waiting until the limits are removed by Google (which was, I think, every 24 hours), or running continuing the queries with a different IP address or using a VPN.

However, now I see that this error message is sometimes produced right away, but this is not consistent. The issue is still not with trendecons package, however. Trendecon depends on gtrendsR to do the queries from google API. And the same issue is discussed there at length: https://github.com/PMassicotte/gtrendsR/issues/431 with some people saying it is persistent, while others saying that everything is working fine for them.

I am not really sure what the answer is why it works for some people and not for others, but most likely it will be something on the Google server side, for which neither we, nor the authors of gtrendsR can do anything about.

[karoliskoncevicius]

However, the final part of your error message:

Error in `mutate()`:
ℹ In argument: `trend_data = list(...)`.
ℹ In row 11.
Caused by error in `curl::curl_fetch_memory()`:
! Timeout was reached: [trends.google.com] Send failure: Connection was reset
Run `rlang::last_trace()` to see where the error occurred.

I was not able to reproduce. If possible, maybe someone can share a keyword which results in this error. Thou from the error it seems like the server reseted the connection, so might not be easily reproducible.

[karoliskoncevicius]

Finally, I think this reponse: https://github.com/trendecon/trendecon/issues/74#issuecomment-1172154343 should answer the question about why values can be negative or go beyond 100.