EE1.14 slow import

[denisivanov]

I am testing Magmi v0.7.21 with Magento EE 1.14 and experiencing very slow import times. 700 products taking about 15 minutes. Database seems maxed out, so I checked for slow queries and found that these are taking several seconds each:

SELECT entity_id FROM catalog_product_entity_varchar WHERE attribute_id='86' AND store_id='0' AND value='shoes-xyz' ORDER BY entity_id

I have been running earlier versions of Magmi and this hasn't been an issue.

I am able to create an idea across all 3 columns, including 'value' , which speeds things up a lot. However, the index is huge. I would like to avoid it, if possible. Is this the right way to proceed, or is there a different solution ?

[dweeves]

The "magmi optimizer" plugin creates the indexes , use it (only once is sufficient). And no, i didn't find any alternative way to speed imports. The thing is magmi is performing "search by value" to find if it has to create a new value for select/multiselect attributes or not.

[denisivanov]

Thanks for reply. I did try the optimizer, but it doesn't see to create index for this particular table: catalog_product_entity_varchar

[dweeves]

OK, this one should not be indexed (too many values). this call seems to list values for attribute 86 in your setup. what is this attribute ? it may help me to find which part of the code is guilty for this unoptimized call

[denisivanov]

That's "url_key" attribute.

The funny thing is that we have EE 1.12 in production with Magmi v0.7.18 and it works great. I tested the query on the production database and it is taking a long time (there is no idea on the value), so i am guessing that the query is not present in v0.7.18

I also noticed that URL is not being indexed properly in EE 1.14 . url_key attribute IS set properly after the import, but it doesn't look like url rewrite is being created/indexed correctly. If I change url_key manually, the rewrite is created properly.

[dweeves]

ok, in that case, that's the url key conflict checker that is in cause. it's a feature that is not in 0.7.18 , but it's an important one since if many items are imported with same url key (or same slugged name), then the product won't be accessible in magento. so i added the url conflict checker. i'll fix this or would add an option to disable this behaviour. thanks for spotting.

[denisivanov]

Thank you.

Do you have any pointers for EE1.14 url_key imports ?

[dweeves]

EE 1.14 added a specific EE table to handle rewrites. i need to find out which table it is. if you have a PHPMyAdmin, send me a link to my email and i'll surely be able to add code into magmi to handle it.

[denisivanov]

I emailed about phpMyAdmin access.

Also, adding product ID to this table seems to fix the url issue: catalog_product_entity_url_key

[dweeves]

latest git should have better performance, reworked url key processing & added specific cpeuk table code