Memory Leak - Sidekiq

[victorlcampos]

Hi,

First of all, thanks for this awsome gem, it's helping a lot here in my project. Second, sorry for my english, it's not my native language and i'm still improving it.

We are using POI (https://github.com/victorlcampos/poi_spreadsheet) to write to Excel with good performance and it's working amazing.

But when I'm using with a Sidekiq, the first time the workers process the file (1MM lines +-) it's consume 85% from my memory machine (ok). But this memory isn't free after perform the job. So, when it run again it shutdown the VM because no memory.

Could you help me understand why?

Here where ai Load POI (that load RJB https://github.com/victorlcampos/poi_spreadsheet/blob/master/lib/poi_spreadsheet.rb)

module Vportal
  class ExcelExport

    def self.copy_to_template(data, template_path, fetch_more_data=nil)
      first_row = data[:first_row] || 0
      first_col = data[:first_col] || 0
      sheet_name = data[:sheet_name]

      book = PoiSpreadsheet.load(template_path, sheet_name, '1024M')

      unless data[:skip_columns]
        book.sheets[sheet_name].set_values(first_row, first_col, data[:columns])
        first_row += 1
      end

      last_row = 0
      data[:values].each_with_index do |row, i|
        book.sheets[sheet_name].set_values(i + first_row, first_col, row)
        last_row += 1
      end

      if fetch_more_data
        while (data = fetch_more_data.call) && !data.blank?
          total_rows = 0

          data.each_with_index do |row, i|
            book.sheets[sheet_name].set_values(i + last_row + first_row, first_col, row)
            total_rows += 1
          end

          last_row += total_rows
        end
      end

      book
    end
  end
end

Here where I save the Excel workbook

         begin
          temp_file = Tempfile.new("#{report.name}.xlsx")

          book = Vportal::ExcelExport.copy_to_template(result_template,
                                                       template_path,
                                                       fetch_more_data(result['_scroll_id'], result_template))
          book.save(temp_file.path)

          attach_to_report(report, temp_file)
        rescue => e
          report.status = :error
          report.error_message = { e.message => e.backtrace }.to_json
          report.save(validate: false)
        ensure
          temp_file.close
          temp_file.unlink
        end

Book Save Method where I think that should remove it from memory:

def save file_name = @file_name
      @file_output_class ||= Rjb::import('java.io.FileOutputStream')
      out = @file_output_class.new(file_name);

      begin
        j_book.write(out)
      ensure
        out.close
        j_book.dispose
        self.j_book = nil
      end
    end

[arton]

Thank you for your report. Because I'm not familiar with Sidekiq, I'd like some more information about this issue. First, if without Sidekiq, the memory leak also happened? For example, I imagine the script such as below:

require 'rjb'
...load POI
...manipulate the objects
...end of the manipulation
gets #stop for waiting 'hit return key'
# at this point, it consumes heavy memory?

If above script consumed many memory, it naturally caused by Rjb's miss handles of freed objects. Then, I wonder if you could copy and paste the script in this issue for my testing. However, if above script does not reproduce the issue, Rjb is not guilty.

BTW Some consideration is, your version of Rjb is the same ruby version in Sidekiq? I have heard about that some trouble was happened by below step.

1. rbenv some version of ruby
2. gem install Rjb (Rjb was compiled with above ruby's version).
3. Sidekiq runs within another version of ruby but Gem loads Rjb from same repositry of 2. => causes ambiguous runtime behavior. If your ruby of Sidekiq and Rjb installed ruby were not same, it should be above cnodition.

Regards

[victorlcampos]

Stale issue I dont remember if it's fixed