IOError On Large Responses

[jkeam]

When the query returns a large response, the client bombs and throws an IOError. This is caused from line 22 of file uri.rb

::URI.parse(uri).read

For large reads, this causes a problem. Perhaps use the open method that lets you process the file in piecemeal instead.

Of course then this will have to refactored as well (client.rb:55):

  else
    response = URI.get(@uri)
    response = %Q|[#{response.split("\n").join(',')}]| if resource == 'export'
    Utils.to_hash(response, @format)
  end

As this tries to process the entire contents at once.

[keolo]

Thanks for the heads up. I'm pretty booked at the moment so it may take a while to resolve this. Would you like to take a stab at it?

[jkeam]

I've looking at it a bit and I can't seem to get the IOError anymore. What I get instead is the client hangs for a very long time until eventually I get a timeout error. You see a similar behavior if you use a regular web browser. The browser spins as it pulls in all that data, but you see the data incrementally as it receives more and more. For the ruby client, b/c it waits for the entire response, you can't start processing until all the data is pulled in.

Below is the cleansed request and the resulting stack trace I got. I'm going to mark this as closed as I can't reproduce the original issue.

Request

https://data.mixpanel.com/api/2.0/export?api_key=XXX&expire=XXX&format=json&from_date=2013-12-01&sig=XXX&to_date=2013-12-31

Stacktrace

        C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/protocol.rb:146:in `rescue in rbuf_fill': Timeout::Error (Timeout::Error)
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/protocol.rb:140:in `rbuf_fill'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:2770:in `read_chunked'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:2750:in `read_body_0'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:2710:in `read_body'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/open-uri.rb:321:in `block (2 levels) in open_http'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:1322:in `block (2 levels) in transport_request'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:2671:in `reading_body'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:1321:in `block in transport_request'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:1293:in `request'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/open-uri.rb:312:in `block in open_http'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/net/http.rb:745:in `start'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/open-uri.rb:306:in `open'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/open-uri.rb:775:in `buffer_open'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/open-uri.rb:203:in `block in open_loop'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/open-uri.rb:201:in `catch'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/open-uri.rb:201:in `open_loop'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/open-uri.rb:146:in `open_uri'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/open-uri.rb:677:in `open'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/1.9.1/open-uri.rb:685:in `read'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/mixpanel_client-3.1.2/lib/mixpanel/uri.rb:22:in `get'
        from C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/mixpanel_client3.1.2/lib/mixpanel/client.rb:55:in`request'