search

gimite / google-drive-ruby

A Ruby library to read/write files/spreadsheets in Google Drive/Docs.
Other
1.76k stars 491 forks source link

Google returning 302 after trying to update many cells #117

Closed thomasdziedzic closed 8 years ago

thomasdziedzic commented 9 years ago

I printed out the result at https://github.com/gimite/google-drive-ruby/blob/master/lib/google_drive/worksheet.rb#L335 after I was getting the exception. It seems to work for a while, and then after updating many cells, it returns the exception.

"<?xml version=\"1.0\"?>\n<HTML>\n<HEAD>\n<TITLE>Moved Temporarily</TITLE>\n</HEAD>\n<BODY BGCOLOR=\"#FFFFFF\" TEXT=\"#000000\">\n<H1>Moved Temporarily</H1>\nThe document has moved <A HREF=\"https://spreadsheets.google.com/feeds/cells/<spreadsheet key>/o5bjv04/private/full/batch?lsrp=1\">here</A>.\n</BODY>\n</HTML>\n"

The stack trace is:

/Users/thomas/.rvm/gems/ruby-1.9.3-p194/gems/nokogiri-1.6.3.1/lib/nokogiri/xml/node.rb:159:in `evaluate': Undefined namespace prefix: //atom:entry (Nokogiri::XML::XPath::SyntaxError)
        from /Users/thomas/.rvm/gems/ruby-1.9.3-p194/gems/nokogiri-1.6.3.1/lib/nokogiri/xml/node.rb:159:in `block in xpath'
        from /Users/thomas/.rvm/gems/ruby-1.9.3-p194/gems/nokogiri-1.6.3.1/lib/nokogiri/xml/node.rb:150:in `map'
        from /Users/thomas/.rvm/gems/ruby-1.9.3-p194/gems/nokogiri-1.6.3.1/lib/nokogiri/xml/node.rb:150:in `xpath'
        from /Users/thomas/.rvm/gems/ruby-1.9.3-p194/gems/nokogiri-1.6.3.1/lib/nokogiri/xml/node.rb:214:in `css'
        from /Users/thomas/.rvm/gems/ruby-1.9.3-p194/gems/google_drive-0.3.11/lib/google_drive/worksheet.rb:338:in `block in save'
        from /Users/thomas/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/set.rb:222:in `block in each'
        from /Users/thomas/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/set.rb:222:in `each_key'
        from /Users/thomas/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/set.rb:222:in `each'
        from /Users/thomas/.rvm/gems/ruby-1.9.3-p194/gems/google_drive-0.3.11/lib/google_drive/worksheet.rb:306:in `each_slice'
        from /Users/thomas/.rvm/gems/ruby-1.9.3-p194/gems/google_drive-0.3.11/lib/google_drive/worksheet.rb:306:in `save'
emerose commented 9 years ago

FWIW, I ran into this problem as well. Google was throwing 302s and 500s regularly, making saving a file of moderate size (~1500 rows) impossible. I ended up using the patch from d99f421 to specify larger chunks, and then adding a bunch of ugly retries on top of that (see 3bc03d1f)

gimite commented 9 years ago

Thanks for the report. You mean, you initially had "Undefined namespace prefix" exception, but somehow it has gone away, and instead encountered 302 issue? Returning 302 seems weird... If you constantly see it, it may be better to report here: https://code.google.com/a/google.com/p/apps-api-issues/issues/list?q=label:API-Spreadsheets

emerose commented 9 years ago

@gimite sort of. If Google throws a 302, the exception you get is "Undefined namespace prefix" since the body of the 302 response doesn't have the atom:entry content. My ugly workaround is just to catch 302 and 500 explicitly and retry: https://github.com/gimite/google-drive-ruby/blob/3bc03d1f4f05ed6b7861770334c38cfcbd64ff80/lib/google_drive/session.rb#L472

I agree that there's some underlying bug in the Spreadsheets API here. Still, retrying once or twice seems to work.

thomasdziedzic commented 9 years ago

Is this possibly something that could be fixed by https://github.com/gimite/google-drive-ruby/issues/27 ?

emerose commented 9 years ago

@gostrc I hope so!

johnnygoodman commented 9 years ago

I would love to see this fixed. I noticed that there was a fork and hoping for a better chunking scheme tried this without success:

gem 'google_drive', :git => 'https://github.com/PagerDuty/google-drive-ruby.git', :branch => 'master'

I got it working by wrapping all my saves in retryable blocks like so:

   retryable(:tries => 5) do
          ws.save
        end
thomasdziedzic commented 9 years ago

@johnnygoodman note, I forked it under the more-chunky branch rather than master: https://github.com/PagerDuty/google-drive-ruby/tree/more-chunky

But it doesn't include retrying, just an increased chunk size.

gimite commented 9 years ago

@emerose Oh I see.

Is this possibly something that could be fixed by #27 ?

It's available as pre version, so you can try it with:

$ gem install --pre google_drive

It's unlikely that switch from Document List API to Drive API affects, but there would be small chance that switch from old version of Spreadsheet API to Spreadsheet API V3 (which is also included in the new version) will fix it.

If it's still not fixed, it may be better to have built-in support of retrying...

johnnygoodman commented 9 years ago

@gostrc noted and thanks for pointing that out.

For kicks I ran my script from the more-chunky branch:

Using google_drive 0.3.11 from https://github.com/PagerDuty/google-drive-ruby.git (at more-chunky)

No joy.

I then added the retryable back and it is confirmed working with both the gimite/master and PagerDuty/more-chunky gems branches.

rubyyan commented 9 years ago

@gimite The --pre google_drive helped me. Thanks

thomasdziedzic commented 9 years ago

@emerose I started getting 500's and 302 even with my chunk size increase. Your branch fixed the issue except for the 500s. I added some code to your code from my own branch to handle 500s: https://github.com/PagerDuty/google-drive-ruby/tree/fix-500

emerose commented 9 years ago

@gostrc Hm, strange. I'm not sure why 2d64445e676158c7970e1d42c975f9ab9a94d7e4 makes a difference. Shouldn't the next on this line https://github.com/PagerDuty/google-drive-ruby/blob/2d64445e676158c7970e1d42c975f9ab9a94d7e4/lib/google_drive/session.rb#L477 skip the rest of the method (with the elsif clause) whenever the if branch gets hit?

-sq

thomasdziedzic commented 9 years ago

@emerose Looking at it again, it looks like it should have. The only reason why it might have been skipped though is if the following returned false due to a request:

params[:retries] && (params[:retries] > 0)

unfortunately I didn't figure out which one it was when I wrote the code.

thomasdziedzic commented 9 years ago

@emerose I had to do some other investigations since we we're getting more failures. After some investigation, it seems like other requests in the save method that don't actually do the save are getting 302s, 400s, and 500s.

gimite commented 9 years ago

I released google_drive 1.0.0.pre2 where chunking of batch request is disabled. If you have encountered 500 or other errors on save, can you try 1.0.0.pre2 to see if it's solved, gets better or gets worse? You can try it with:

$ sudo gem install --pre google_drive
dustinsgoodman commented 9 years ago

Issue isn't resolved and appears to be worse in the pre2 build. I'm batching in sets of 15 rows with ~30 columns a piece. Here's the output from my console:

/Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/nokogiri-1.6.2.1/lib/nokogiri/xml/node.rb:159:in `evaluate': Undefined namespace prefix: //atom:entry (Nokogiri::XML::XPath::SyntaxError)
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/nokogiri-1.6.2.1/lib/nokogiri/xml/node.rb:159:in `block in xpath'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/nokogiri-1.6.2.1/lib/nokogiri/xml/node.rb:150:in `map'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/nokogiri-1.6.2.1/lib/nokogiri/xml/node.rb:150:in `xpath'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/nokogiri-1.6.2.1/lib/nokogiri/xml/node.rb:214:in `css'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/google_drive-0.3.11/lib/google_drive/worksheet.rb:336:in `block in save'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/2.1.0/set.rb:263:in `each_key'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/2.1.0/set.rb:263:in `each'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/google_drive-0.3.11/lib/google_drive/worksheet.rb:306:in `each_slice'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/google_drive-0.3.11/lib/google_drive/worksheet.rb:306:in `save'
    from script/stats-scripts/cybermonday.rb:85:in `block in <top (required)>'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activerecord-4.0.4/lib/active_record/relation/delegation.rb:63:in `each'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activerecord-4.0.4/lib/active_record/relation/delegation.rb:63:in `each_slice'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activerecord-4.0.4/lib/active_record/relation/delegation.rb:63:in `method_missing'
    from script/stats-scripts/cybermonday.rb:37:in `<top (required)>'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/railties-4.0.4/lib/rails/commands/runner.rb:51:in `eval'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/railties-4.0.4/lib/rails/commands/runner.rb:51:in `<top (required)>'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activesupport-4.0.4/lib/active_support/dependencies.rb:229:in `require'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activesupport-4.0.4/lib/active_support/dependencies.rb:229:in `block in require'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activesupport-4.0.4/lib/active_support/dependencies.rb:214:in `load_dependency'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activesupport-4.0.4/lib/active_support/dependencies.rb:229:in `require'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/railties-4.0.4/lib/rails/commands.rb:84:in `<top (required)>'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activesupport-4.0.4/lib/active_support/dependencies.rb:229:in `require'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activesupport-4.0.4/lib/active_support/dependencies.rb:229:in `block in require'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activesupport-4.0.4/lib/active_support/dependencies.rb:214:in `load_dependency'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activesupport-4.0.4/lib/active_support/dependencies.rb:229:in `require'
    from /Users/dustingoodman/Documents/springbot/bin/rails:8:in `<top (required)>'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activesupport-4.0.4/lib/active_support/dependencies.rb:223:in `load'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activesupport-4.0.4/lib/active_support/dependencies.rb:223:in `block in load'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activesupport-4.0.4/lib/active_support/dependencies.rb:214:in `load_dependency'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/activesupport-4.0.4/lib/active_support/dependencies.rb:223:in `load'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from /Users/dustingoodman/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from -e:1:in `<main>'
gimite commented 9 years ago

Hmm outputting that error is weird. I have also modified the code to raise ResponseCodeError instead for 3xx HTTP response code.

Can you check what is HTTP response code and response body in that case? You can check it by printing response.code and response.body at: https://github.com/gimite/google-drive-ruby/blob/drive_api/lib/google_drive/session.rb#L420

Also, can you create a new spreadsheet in drive.google.com and try the code with the new spreadsheet? I'm wondering if it is specific to old spreadsheets.

dustinsgoodman commented 9 years ago

Don't know what's happened in the last 3 weeks but creating a new spreadsheet definitely helped. Weirdness....

apsey commented 9 years ago

FYI: I run a job that puts lots of information on a google spreadsheet. A month ago or so, I started receiving emails from cron daemon saying that my job failed. As soon as I saw the error, I reprogrammed my crontab and runned successfully on the very same day, the same requests on a new spreadsheet.

I just used the command "gem install --pre google_drive" and tomorrow I will notify you if that did the fix.

apsey commented 9 years ago

It didnt work for me, I guess i will try the google-api-client gem

gimite commented 9 years ago

@apsey What was the error message?

BMayneFactual commented 9 years ago

for what it's worth, gem install --pre seems to be working for me for ~300 rows at a time.