jennybc / googlesheets

Google Spreadsheets R API
Other
783 stars 191 forks source link

does gs4_create() support writing more than 200,000 rows of data? #400

Closed aaronmams closed 4 years ago

aaronmams commented 4 years ago

I can't seem to use the gs4_create() function to move more than 200,000 rows of data. Here is the error I am receiving:

n<-200000
> test.df <- data.frame(x1=rnorm(n),x2=rnorm(n),x3=rnorm(n),x4=rnorm(n),x5=rnorm(n))
> t <- Sys.time()  
> gs4_create("test-data3",
+            sheets=test.df
+ )
Creating new Sheet: "test-data3"
Error: Client error: (400) INVALID_ARGUMENT
  * Client specified an invalid argument. Check error message and error details for more information.
  * Invalid requests[0].updateSheetProperties: This action would increase the number of cells in the workbook above the limit of 5000000 cells.

What makes me suspect a row limitation somewhere is that I can successfully execute the same code with only slightly fewer rows:

n<-180000
> test.df <- data.frame(x1=rnorm(n),x2=rnorm(n),x3=rnorm(n),x4=rnorm(n),x5=rnorm(n))
> t <- Sys.time()  
> gs4_create("test-data3",
+            sheets=test.df
+ )
Creating new Sheet: "test-data3"
> Sys.time() - t
Time difference of 3.207702 mins
> object.size(test.df)
7201224 bytes

And even trying to upload a data frame of much larger size (but fewer than 200k rows) doesn't seem to be problematic:

n<-180000
test.df <- data.frame(x1=rnorm(n),x2=rnorm(n),x3=rnorm(n),x4=rnorm(n),x5=rnorm(n),
                      x6=runif(n),x7=runif(n),x8=runif(n),x9=runif(n),x10=runif(n))
t <- Sys.time()  
gs4_create("test-data3",
           sheets=test.df
)
Sys.time() - t
Time difference of 7.456010 mins
object.size(test.df)
12001904 bytes

I've read about Google's limitation of 5,000,000 cells (rows X columns) but my 200k rows by 5 columns wouldn't exceed this...also the "Client Error" makes me think this has something to do with me and not with Google.

Any advice on how to fix (or further debug this)?

Thanks.