search

magento-engcom / import-export-improvements

Open Software License 3.0
31 stars 29 forks source link

Huge memory usage when importing large quantities of products #92

Open piotrekkaminski opened 6 years ago

piotrekkaminski commented 6 years ago

imported from: https://github.com/magento/magento2/issues/10914

When importing huge amounts of products Magento 2 is loading contents of importexport_importdata to memory and then iterate it in memory which results in huge memory usage

Preconditions

  1. Magento 2.1.7
  2. CSV with 1 500 000 products

Steps to reproduce

  1. Validate 1 500 000 products
  2. Run import

Expected result

  1. Memory usage for iterating trough import data should be constant

Actual result

  1. Whole table contents is loaded to memory
piotrekkaminski commented 6 years ago

Pull request: https://github.com/magento/magento2/pull/10918

piotrekkaminski commented 6 years ago

From @magento-engcom-team: @diwipl, thank you for your report. We've created internal ticket(s) MAGETWO-84592 to track progress on the issue.

nemesis-back commented 6 years ago

@dmanners @piotrekkaminski assign to me please

dmanners commented 6 years ago

Hi @nemesis-back thank you for reaching out to us. I have sent you and invitation to the github team. Once you have accepted this invite you will have permissions to assign a task to yourself. If you need any further help feel free to reach out to me.

dmanners commented 5 years ago

Hi @nemesis-back did you get anything working on this? Or have any debugging information that might be useful for people to continue working with?