search

keolo / mixpanel_client

Ruby interface to the Mixpanel Data API
MIT License
148 stars 72 forks source link

NoMemoryError when exporting huge events #41

Closed wnkz closed 1 year ago

wnkz commented 9 years ago

Hi, I'm having a problem when I try to export a very large number of events

client.request('export', from_date: from_date, to_date: to_date).each do |event|
  file.puts(event.to_json)
end

We get:

2015-04-09T16:46:00.705Z 1 TID-otovfis9w WARN: failed to allocate memory
2015-04-09T16:46:00.705Z 1 TID-otovfis9w WARN: /usr/src/app/vendor/bundle/ruby/2.2.0/gems/mixpanel_client-4.1.1/lib/mixpanel/client.rb:69:in `make_normal_request'

I think it's because the HTTP body of the response is to big to be stored in memory at once. There should be a way to stream or chunk the output.

What do you think ?

keolo commented 9 years ago

That seems reasonable. Would you like to take a stab at it?