Image attributes import is too slow

[yutv]

Related issue #146

Environment and configuration

1. Magento Cloud 2.3.4
2. akeneo/module-magento2-connector-community 100.4.7
3. PHP 7.2
4. 467 images on Akeneo

   ```sql SELECT COUNT(*) as cnt FROM ( SELECT images_2 FROM tmp_akeneo_connector_entities_product UNION SELECT images_3 FROM tmp_akeneo_connector_entities_product UNION SELECT images_4 FROM tmp_akeneo_connector_entities_product UNION SELECT images_5 FROM tmp_akeneo_connector_entities_product UNION SELECT images_6 FROM tmp_akeneo_connector_entities_product UNION SELECT images_7 FROM tmp_akeneo_connector_entities_product UNION SELECT images_10 FROM tmp_akeneo_connector_entities_product UNION SELECT images_11 FROM tmp_akeneo_connector_entities_product ) as tmp; ``` ``` +-----+ | cnt | +-----+ | 467 | +-----+ ``` * this query is executed when the "Import image attributes" message shown on the screen

Steps to reproduce

1. start import

Expected result

1. Reasonable time for images import, e.g. 5-10 mins

Actual result

1. Images import took about 1h45m, it is about 5 images per minute, see log:

   [10:06:26] Import image attributes
   [11:50:06] Drop temporary table

   Code

There is three levels loop in the source code. It is definitively a bottleneck. Can it be optimized?

Loop 1: https://github.com/akeneo/magento2-connector-community/blob/404e213e4abf17191ee4d525dbf156ab871f58d4/Job/Product.php#L2453 Loop 2: https://github.com/akeneo/magento2-connector-community/blob/404e213e4abf17191ee4d525dbf156ab871f58d4/Job/Product.php#L2456 Loop 3: https://github.com/akeneo/magento2-connector-community/blob/404e213e4abf17191ee4d525dbf156ab871f58d4/Job/Product.php#L2514

P.S.: Also adding usage of the https://aria2.github.io/ utility may dramatically improve image downloading. Of course, aria2 usage must be optional and disabled by default because it requires to install additional software on the hosting.

[wojmas]

@yutv while you waited 1h have you got an error: A technical problem with the server created an error. Try again to continue what you were doing. If the problem persists, try again later. ??

[yutv]

@wojmas no, there was no such error.

Akeneo log is the following:

[09:38:14] Import image attributes
[10:26:15] Drop temporary table
[10:26:15] completed

No "technical problem" error in the va/log/*.log files.

[iizno]

@yutv Did you find any solution to speed up the connector ? We have the same issue.

[yutv]

Hello @iizno

The only optimization I use is native options Stores > Configuration > Catalog > Akeneo Connector > Filter Products

1. Updated mode = Since last X days
2. Updated = 1

screenshot


It allows to do a kind of incremental import instead of a full one .

But sometimes it's necessary reimport all products. Unfortunately I have no solution for such case, so usually it takes about 18 hours to import 20,000 products.

[madleen]

Hello, I'm reopening this topic because I also encounter a lot of latency on image imports.

[17:11:06] Start import
...
[17:11:56] 2576 line(s) found
...
[17:18:08] Import image attributes
[18:48:50] completed
...

Environment and configuration

Magento Community : 2.4.5-p1 akeneo/module-magento2-connector-community : 103.3.1 PHP : 8.1 MySQL : 8

Context:

Websites : 5 Stores view : 9 Total products : 12757

Configuration:

After some analysis, I realized that I was doing 5185 insertOnDuplicate for 1 single product and there are several identical queries that are executed 24 times. 😨 https://github.com/akeneo/magento2-connector-community/blob/v103.3.1/Job/Product.php#L3986

Mysql 8 no longer embeds a cache so we are directly impacted. I also have the impression that the IF does not seem to work, I never go into it during my debug. Do you think it would be possible to improve this image import management?

Has anyone found an alternative to remedy this problem of slowness? I was thinking of testing MariaDB in the meantime, in order to test the cache system. Good Day, Madleen