search

magento-engcom / import-export-improvements

Open Software License 3.0
31 stars 29 forks source link

Memory error on customer import #97

Closed dmanners closed 6 years ago

dmanners commented 6 years ago

Description

With the following settings a customer import would crash with an out of memory problem.

  1. Current database of 250k customers (can be created using bin/magento setup:perf:generate-fixtures setup/performance-toolkit/profiles/ce/small.xml and editing the small.xml to setup 250000 customers)
  2. Memory limit set via .htaccess to 268M

Fixed Issues (if relevant)

  1. magento-engcom/import-export-improvements#43: Failure to allocate memory error when importing customers in admin

Manual testing scenarios

  1. After setting up the database and memory limit as described visit the admin section and try to import at least 1 customer,

Contribution checklist

dmanners commented 6 years ago

I have some open questions on this solution I would like to get people's thoughts on.

There are two approaches here either replace the usage of CollectionByPagesIteratorFactory inside this class or try to get the iterator to work without making objects. I picked to replace the usage here as the iterator is used in not just the customer import but other import processes, though that does mean this problem can happen in multiple places. Should we fix or replace?

The current Storage class allows you to init it with a page size. My current changes ignores this and just loads all the customers. I am not too sure in what cases page size is setup but it maybe that we find some cases that only load a limited set of customers during import or export.

dmanners commented 6 years ago

Also I am still working on the tests but I thought I would get people's thoughts first.

magento-engcom-team commented 6 years ago

@dmanners thank you for contributing. Please accept Community Contributors team invitation here to gain extended permissions for this repository.