search

MarkEdmondson1234 / searchConsoleR

R interface with Google Search Console API v3, including Search Analytics.
http://code.markedmondson.me/searchConsoleR/
Other
114 stars 41 forks source link

Error: Error in x[[1]] : subscript out of bounds #62

Open AdamJayJonca opened 3 years ago

AdamJayJonca commented 3 years ago

When getting the request by date: 

search_analytics(`my domain`,
                 "2020-07-29", 
                 Sys.Date()-4, 
                 dimensions =  c(
                     "date",
                     "country",  
                     "query"
                 ),
                  walk_data = "byDate"
)

I get the error: Error in x[[1]] : subscript out of bounds Calls: sourceWithProgress ... search_analytics -> gar_batch_walk -> Reduce -> lapply -> FUN The output when the walk_data is byBatch is proper.

options(googleAuthR.verbose=2):

Fetching search analytics for url: xxxxx dates: 2020-07-29 2020-08-01 dimensions: date country query dimensionFilterExp:  searchType: web aggregationType: auto
Batching data via method: byDate
Will fetch up to 25000 rows per day
i 2020-08-04 15:20:01 > Batch API limited to [ 1 ] calls at once.
i 2020-08-04 15:20:01 > Request #:  2020-07-29
i 2020-08-04 15:20:01 > Constructing batch request URL for:  /webmasters/v3/sites/xxxxx/searchAnalytics/query
i 2020-08-04 15:20:02 > Making new batched cache
i 2020-08-04 15:20:02 > Making Batch API call
i 2020-08-04 15:20:04 > Request #:  2020-07-30
i 2020-08-04 15:20:04 > Constructing batch request URL for:  /webmasters/v3/sites/xxxxx/searchAnalytics/query
i 2020-08-04 15:20:04 > Making new batched cache
i 2020-08-04 15:20:04 > Making Batch API call
i 2020-08-04 15:20:07 > Request #:  2020-07-31
i 2020-08-04 15:20:07 > Constructing batch request URL for:  /webmasters/v3/sites/xxxxx/searchAnalytics/query
i 2020-08-04 15:20:07 > Making new batched cache
i 2020-08-04 15:20:07 > Making Batch API call
i 2020-08-04 15:20:09 > Request #:  2020-08-01
i 2020-08-04 15:20:09 > Constructing batch request URL for:  /webmasters/v3/sites/xxxx/searchAnalytics/query
i 2020-08-04 15:20:09 > Making new batched cache
i 2020-08-04 15:20:09 > Making Batch API call
Error in value[[3L]](cond) : 
  Error with batch response - writing response to C:\Users\A7D10~1\AppData\Local\Temp\RtmpeixfLX\filef584e4a786e.rds

Session Info

`R version 3.6.3 (2020-02-29) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 18363)

Matrix products: default

Random number generation: RNG: Mersenne-Twister Normal: Inversion Sample: Rounding

locale: [1] LC_COLLATE=Polish_Poland.1250 LC_CTYPE=Polish_Poland.1250 LC_MONETARY=Polish_Poland.1250 LC_NUMERIC=C LC_TIME=Polish_Poland.1250

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] forcats_0.5.0 stringr_1.4.0 dplyr_1.0.0 purrr_0.3.4 readr_1.3.1 tidyr_1.1.0 tibble_3.0.3 ggplot2_3.3.2 tidyverse_1.3.0
[10] searchConsoleR_0.4.0 googleAnalyticsR_0.8.0 googleAuthR_1.3.0 RJDBC_0.2-8 rJava_0.9-11 DBI_1.1.0

loaded via a namespace (and not attached): [1] tidyselect_1.1.0 slam_0.1-47 NLP_0.2-0 haven_2.2.0 gargle_0.4.0 lattice_0.20-38 colorspace_1.4-1 vctrs_0.3.1 generics_0.0.2 yaml_2.2.1
[11] utf8_1.1.4 rlang_0.4.7 pillar_1.4.6 glue_1.4.1 withr_2.1.2 dbplyr_1.4.2 readxl_1.3.1 modelr_0.1.6 lifecycle_0.2.0 plyr_1.8.6
[21] cellranger_1.1.0 munsell_0.5.0 gtable_0.3.0 rvest_0.3.5 memoise_1.1.0 tm_0.7-7 parallel_3.6.3 curl_4.3 fansi_0.4.1 broom_0.5.5
[31] Rcpp_1.0.5 openssl_1.4.2 scales_1.1.0 backports_1.1.5 jsonlite_1.7.0 fs_1.3.1 rCharts_0.4.5 hms_0.5.3 askpass_1.1 digest_0.6.25
[41] stringi_1.4.6 RJSONIO_1.3-1.4 grid_3.6.3 cli_2.0.2 tools_3.6.3 magrittr_1.5 crayon_1.3.4 whisker_0.4 pkgconfig_2.0.3 ellipsis_0.3.1
[51] xml2_1.3.2 reprex_0.3.0 lubridate_1.7.4 assertthat_0.2.1.9000 httr_1.4.1 rstudioapi_0.11 R6_2.4.1 nlme_3.1-144 compiler_3.6.3
`

MarkEdmondson1234 commented 3 years ago

I guess that day had no data, but could you load the contents of the file it writes (C:\Users\A7D10~1\AppData\Local\Temp\RtmpeixfLX\filef584e4a786e.rds in your example) using readRDS(file) and give the contents of that R object here? It should confirm the response structure.

AdamJayJonca commented 3 years ago

I guess that day had no data, but could you load the contents of the file it writes (C:\Users\A7D10~1\AppData\Local\Temp\RtmpeixfLX\filef584e4a786e.rds in your example) using readRDS(file) and give the contents of that R object here? It should confirm the response structure.

The date range have some data (even for sucha a specific request). 'Try this API' tool response shows the proper data: 

curl --request POST \
  'https://www.googleapis.com/webmasters/v3/sites/xxxxxx/searchAnalytics/query?key=[YOUR_API_KEY]' \
  --header 'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --data '{"startDate":"2020-07-29","endDate":"2020-08-01","dimensions":["country","query","date"]}' \
  --compressed
HTTP/1.1 200 
cache-control: private
content-encoding: gzip
content-length: 10616
content-type: application/json; charset=UTF-8
date: Wed, 05 Aug 2020 06:13:29 GMT
server: ESF
vary: Origin, X-Origin, Referer

{
  "rows": [
    {
      "keys": [
        "aut",
        "1sfa611620r1001",
        "2020-07-30"
      ],
      "clicks": 1,
      "impressions": 2,
      "ctr": 0.5,
      "position": 4
    },
MarkEdmondson1234 commented 3 years ago

Can you load the rds file as it’s a batched call and the response needs to be examined.

From: AdamJayJonca notifications@github.com Sent: Wednesday, August 5, 2020 8:19:21 AM To: MarkEdmondson1234/searchConsoleR searchConsoleR@noreply.github.com Cc: Mark github@markedmondson.me; Comment comment@noreply.github.com Subject: Re: [MarkEdmondson1234/searchConsoleR] Error: Error in x[[1]] : subscript out of bounds (#62)

I guess that day had no data, but could you load the contents of the file it writes (C:\Users\A7D10~1\AppData\Local\Temp\RtmpeixfLX\filef584e4a786e.rds in your example) using readRDS(file) and give the contents of that R object here? It should confirm the response structure.

The date range have some data (even for sucha a specific request). 'Try this API' tool response shoe the proper data:

curl --request POST \ 'https://www.googleapis.com/webmasters/v3/sites/xxxxxx/searchAnalytics/query?key=[YOUR_API_KEY]' \ --header 'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --data '{"startDate":"2020-07-29","endDate":"2020-08-01","dimensions":["country","query","date"]}' \ --compressed

HTTP/1.1 200 cache-control: private content-encoding: gzip content-length: 10616 content-type: application/json; charset=UTF-8 date: Wed, 05 Aug 2020 06:13:29 GMT server: ESF vary: Origin, X-Origin, Referer

{ "rows": [ { "keys": [ "aut", "1sfa611620r1001", "2020-07-30" ], "clicks": 1, "impressions": 2, "ctr": 0.5, "position": 4 },

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/MarkEdmondson1234/searchConsoleR/issues/62#issuecomment-669003799, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAYCPLAKEIUVPEJHQMC5N23R7D2WTANCNFSM4PUMYJLA.

AdamJayJonca commented 3 years ago

Can you load the rds file as it’s a batched call and the response needs to be examined.

file.zip

MarkEdmondson1234 commented 3 years ago

This may be related to https://github.com/MarkEdmondson1234/searchConsoleR/issues/43 but I can replicate it

MarkEdmondson1234 commented 3 years ago

This is a subtle problem back in googleAuthR and compounded by the search console API which seems to have variable output, sometimes with leading blank spaces and not which makes it tricky to debug. For now I'd say avoid byDate until it can be sorted out, I'll work on it on this branch https://github.com/MarkEdmondson1234/googleAuthR/tree/batch_indexing

mmueller-cm commented 3 years ago

Hey @MarkEdmondson1234 - Fantastic package! Is there any update to the byDate batching? I'm seeing the same issue on my end. It appears to query the requested dates and then fails out

ℹ 2020-10-16 15:38:20 > Request #: 2020-10-06 ℹ 2020-10-16 15:38:22 > Request #: 2020-10-07 ℹ 2020-10-16 15:38:24 > Request #: 2020-10-08 ℹ 2020-10-16 15:38:25 > Request #: 2020-10-09 ℹ 2020-10-16 15:38:27 > Request #: 2020-10-10 ℹ 2020-10-16 15:38:29 > Request #: 2020-10-11 ℹ 2020-10-16 15:38:30 > Request #: 2020-10-12 ℹ 2020-10-16 15:38:32 > Request #: 2020-10-13 Error in x[[1]] : subscript out of bounds

byBatch executes fine.

MarkEdmondson1234 commented 3 years ago

I think only use byBatch for now - the api itself is undergoing changes so I'm waiting for those to settle before publishing any new versions. It could be byDate is not needed anymore.