search

MarkEdmondson1234 / searchConsoleR

R interface with Google Search Console API v3, including Search Analytics.
http://code.markedmondson.me/searchConsoleR/
Other
114 stars 41 forks source link

Error in rbind for Country queries #46

Open vanderWin opened 6 years ago

vanderWin commented 6 years ago

What goes wrong

Binding results for search analytics fails if there's a day when there is not data at all. Could be that days with no data also won't have countryName data and might therefore skip a column?

Steps to reproduce the problem

Narrowing date ranges confirms that results are parsed properly for days where there is data, it just seems to be an issue for queries which trigger a "No data found for supplied dates - returning NA" response.

Additionally, this never occurs over the same date ranges querying for other dimensions such as Page or Device, this only happens for Country (which returns 7 columns instead of 6).

Expected output

Outputs that do function correctly follow the 7 headers of:

date | country | countryName | clicks | impressions | ctr | position

E.g.

date country countryName clicks impressions ctr position
17/04/2017 are United Arab Emirates 0 3 0 257.6667
17/04/2017 aus Australia 0 9 0 141.1111
17/04/2017 bgd Bangladesh 0 4 0 85.25
17/04/2017 bra Brazil 0 2 0 538.5
17/04/2017 bwa Botswana 0 1 0 336
17/04/2017 can Canada 0 7 0 188.1429

Actual output

start <- Sys.Date() - 80
end <- Sys.Date() - 30

country9 <- search_analytics(siteURL = website,
                              startDate = start,
                              endDate = end,
                              dimensions = download_dimensions,
                              searchType = type,
                              walk_data = c('byDate'))
Fetching search analytics for url: http://www.WEBSITE/ dates: 2018-06-05 2018-07-25 dimensions: date country dimensionFilterExp:  searchType: image aggregationType: auto
Batching data via method: byDate
Will fetch up to 5000 rows per day
2018-08-24 10:12:41> Batch API limited to [1] calls at once.
2018-08-24 10:12:41> Request #: 2018-06-05
2018-08-24 10:12:41> Token exists.
2018-08-24 10:12:41> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:12:41> Making Batch API call
2018-08-24 10:12:43> 500 type error in response
2018-08-24 10:12:43> Request #: 2018-06-06
2018-08-24 10:12:43> Token exists.
2018-08-24 10:12:43> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:12:43> Making Batch API call
2018-08-24 10:12:44> 400 type error in response
2018-08-24 10:12:44> Request #: 2018-06-07
2018-08-24 10:12:44> Token exists.
2018-08-24 10:12:44> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:12:44> Making Batch API call
2018-08-24 10:12:46> Request #: 2018-06-08
2018-08-24 10:12:46> Token exists.
2018-08-24 10:12:46> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:12:46> Making Batch API call
2018-08-24 10:12:47> Request #: 2018-06-09
2018-08-24 10:12:47> Token exists.
2018-08-24 10:12:47> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:12:47> Making Batch API call
2018-08-24 10:12:48> Request #: 2018-06-10
2018-08-24 10:12:48> Token exists.
2018-08-24 10:12:48> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:12:48> Making Batch API call
2018-08-24 10:12:50> Request #: 2018-06-11
2018-08-24 10:12:50> Token exists.
2018-08-24 10:12:50> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:12:50> Making Batch API call
2018-08-24 10:12:51> 400 type error in response
2018-08-24 10:12:51> Request #: 2018-06-12
2018-08-24 10:12:51> Token exists.
2018-08-24 10:12:51> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:12:51> Making Batch API call
2018-08-24 10:12:53> Request #: 2018-06-13
2018-08-24 10:12:53> Token exists.
2018-08-24 10:12:53> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:12:53> Making Batch API call
2018-08-24 10:12:54> Request #: 2018-06-14
2018-08-24 10:12:54> Token exists.
2018-08-24 10:12:54> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:12:54> Making Batch API call
2018-08-24 10:12:55> 500 type error in response
2018-08-24 10:12:55> Request #: 2018-06-15
2018-08-24 10:12:55> Token exists.
2018-08-24 10:12:55> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:12:55> Making Batch API call
2018-08-24 10:12:57> Request #: 2018-06-16
2018-08-24 10:12:57> Token exists.
2018-08-24 10:12:57> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:12:57> Making Batch API call
2018-08-24 10:12:58> 500 type error in response
2018-08-24 10:12:58> Request #: 2018-06-17
2018-08-24 10:12:58> Token exists.
2018-08-24 10:12:58> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:12:58> Making Batch API call
2018-08-24 10:12:59> 400 type error in response
2018-08-24 10:12:59> Request #: 2018-06-18
2018-08-24 10:12:59> Token exists.
2018-08-24 10:12:59> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:12:59> Making Batch API call
2018-08-24 10:13:01> Request #: 2018-06-19
2018-08-24 10:13:01> Token exists.
2018-08-24 10:13:01> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:01> Making Batch API call
2018-08-24 10:13:02> Request #: 2018-06-20
2018-08-24 10:13:02> Token exists.
2018-08-24 10:13:02> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:02> Making Batch API call
2018-08-24 10:13:04> Request #: 2018-06-21
2018-08-24 10:13:04> Token exists.
2018-08-24 10:13:04> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:04> Making Batch API call
2018-08-24 10:13:05> 400 type error in response
2018-08-24 10:13:05> Request #: 2018-06-22
2018-08-24 10:13:05> Token exists.
2018-08-24 10:13:05> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:05> Making Batch API call
2018-08-24 10:13:07> Request #: 2018-06-23
2018-08-24 10:13:07> Token exists.
2018-08-24 10:13:07> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:07> Making Batch API call
2018-08-24 10:13:08> Request #: 2018-06-24
2018-08-24 10:13:08> Token exists.
2018-08-24 10:13:08> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:08> Making Batch API call
2018-08-24 10:13:11> 500 type error in response
2018-08-24 10:13:11> Request #: 2018-06-25
2018-08-24 10:13:11> Token exists.
2018-08-24 10:13:11> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:11> Making Batch API call
2018-08-24 10:13:12> Request #: 2018-06-26
2018-08-24 10:13:12> Token exists.
2018-08-24 10:13:12> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:12> Making Batch API call
2018-08-24 10:13:13> Request #: 2018-06-27
2018-08-24 10:13:13> Token exists.
2018-08-24 10:13:13> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:13> Making Batch API call
2018-08-24 10:13:15> Request #: 2018-06-28
2018-08-24 10:13:15> Token exists.
2018-08-24 10:13:15> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:15> Making Batch API call
2018-08-24 10:13:16> Request #: 2018-06-29
2018-08-24 10:13:16> Token exists.
2018-08-24 10:13:16> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:17> Making Batch API call
2018-08-24 10:13:18> Request #: 2018-06-30
2018-08-24 10:13:18> Token exists.
2018-08-24 10:13:18> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:18> Making Batch API call
2018-08-24 10:13:19> Request #: 2018-07-01
2018-08-24 10:13:19> Token exists.
2018-08-24 10:13:19> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:19> Making Batch API call
2018-08-24 10:13:21> Request #: 2018-07-02
2018-08-24 10:13:21> Token exists.
2018-08-24 10:13:21> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:21> Making Batch API call
2018-08-24 10:13:22> Request #: 2018-07-03
2018-08-24 10:13:22> Token exists.
2018-08-24 10:13:22> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:22> Making Batch API call
2018-08-24 10:13:23> Request #: 2018-07-04
2018-08-24 10:13:23> Token exists.
2018-08-24 10:13:23> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:23> Making Batch API call
2018-08-24 10:13:25> Request #: 2018-07-05
2018-08-24 10:13:25> Token exists.
2018-08-24 10:13:25> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:25> Making Batch API call
2018-08-24 10:13:26> 500 type error in response
2018-08-24 10:13:26> Request #: 2018-07-06
2018-08-24 10:13:26> Token exists.
2018-08-24 10:13:26> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:26> Making Batch API call
2018-08-24 10:13:28> Request #: 2018-07-07
2018-08-24 10:13:28> Token exists.
2018-08-24 10:13:28> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:28> Making Batch API call
2018-08-24 10:13:29> Request #: 2018-07-08
2018-08-24 10:13:29> Token exists.
2018-08-24 10:13:29> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:29> Making Batch API call
2018-08-24 10:13:30> Request #: 2018-07-09
2018-08-24 10:13:30> Token exists.
2018-08-24 10:13:30> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:30> Making Batch API call
2018-08-24 10:13:32> Request #: 2018-07-10
2018-08-24 10:13:32> Token exists.
2018-08-24 10:13:32> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:32> Making Batch API call
2018-08-24 10:13:33> Request #: 2018-07-11
2018-08-24 10:13:33> Token exists.
2018-08-24 10:13:33> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:33> Making Batch API call
2018-08-24 10:13:35> Request #: 2018-07-12
2018-08-24 10:13:35> Token exists.
2018-08-24 10:13:35> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:35> Making Batch API call
2018-08-24 10:13:36> Request #: 2018-07-13
2018-08-24 10:13:36> Token exists.
2018-08-24 10:13:36> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:36> Making Batch API call
2018-08-24 10:13:37> Request #: 2018-07-14
2018-08-24 10:13:37> Token exists.
2018-08-24 10:13:37> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:37> Making Batch API call
2018-08-24 10:13:38> Request #: 2018-07-15
2018-08-24 10:13:38> Token exists.
2018-08-24 10:13:38> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:38> Making Batch API call
2018-08-24 10:13:40> Request #: 2018-07-16
2018-08-24 10:13:40> Token exists.
2018-08-24 10:13:40> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:40> Making Batch API call
2018-08-24 10:13:41> Request #: 2018-07-17
2018-08-24 10:13:41> Token exists.
2018-08-24 10:13:41> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:41> Making Batch API call
2018-08-24 10:13:43> Request #: 2018-07-18
2018-08-24 10:13:43> Token exists.
2018-08-24 10:13:43> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:43> Making Batch API call
2018-08-24 10:13:44> Request #: 2018-07-19
2018-08-24 10:13:44> Token exists.
2018-08-24 10:13:44> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:44> Making Batch API call
2018-08-24 10:13:45> Request #: 2018-07-20
2018-08-24 10:13:45> Token exists.
2018-08-24 10:13:45> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:45> Making Batch API call
2018-08-24 10:13:46> Request #: 2018-07-21
2018-08-24 10:13:46> Token exists.
2018-08-24 10:13:46> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:46> Making Batch API call
2018-08-24 10:13:48> Request #: 2018-07-22
2018-08-24 10:13:48> Token exists.
2018-08-24 10:13:48> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:48> Making Batch API call
2018-08-24 10:13:49> 400 type error in response
2018-08-24 10:13:49> Request #: 2018-07-23
2018-08-24 10:13:49> Token exists.
2018-08-24 10:13:49> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:49> Making Batch API call
2018-08-24 10:13:51> 500 type error in response
2018-08-24 10:13:51> Request #: 2018-07-24
2018-08-24 10:13:51> Token exists.
2018-08-24 10:13:51> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:51> Making Batch API call
2018-08-24 10:13:52> Request #: 2018-07-25
2018-08-24 10:13:52> Token exists.
2018-08-24 10:13:52> Constructing batch request URL for: /webmasters/v3/sites/http%3A%2F%2Fwww.WEBSITE%2F/searchAnalytics/query
2018-08-24 10:13:52> Making Batch API call
Error in rbind(deparse.level, ...) : 
  numbers of columns of arguments do not match
In addition: Warning messages:
1: No data found for supplied dates - returning NA 
2: No data found for supplied dates - returning NA 
3: No data found for supplied dates - returning NA 
4: No data found for supplied dates - returning NA 
5: No data found for supplied dates - returning NA

Session Info

R version 3.5.0 (2018-04-23)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252    LC_MONETARY=English_United Kingdom.1252
[4] LC_NUMERIC=C                            LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] jsonlite_1.5         dplyr_0.7.6          searchConsoleR_0.3.0 googleAuthR_0.6.3   

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.17     bindr_0.1.1      magrittr_1.5     tidyselect_0.2.4 R6_2.2.2         rlang_0.2.1      stringr_1.3.1   
 [8] httr_1.3.1       tools_3.5.0      openssl_1.0.1    yaml_2.1.19      assertthat_0.2.0 digest_0.6.15    tibble_1.4.2    
[15] bindrcpp_0.2.2   later_0.7.3      purrr_0.2.5      promises_1.0.1   curl_3.2         memoise_1.1.0    glue_1.2.0      
[22] stringi_1.1.7    compiler_3.5.0   pillar_1.2.3     httpuv_1.4.5     pkgconfig_2.0.1
MarkEdmondson1234 commented 6 years ago

Thank you for the report, yes this is a bug where it can't rbind if the columns are missing for days that have no data. I will look at a safer rbind such as dplyr::bind_rows() or improving the resulting data.frame if no results are found.