Problem between data-definition and pimcore connector for magento

[The-Creeps]

Hello, i have one problem between data-definition and pimcore connector for magento.

data-definition plugin execute in loop the sync function of pimcore connector when i update or insert published product or if i want to publish during import.

When I update from the Pimcore panel, only one request is sent to magento, but when I update from import with definition of data, I can see hundreds of shipments of the same article per second received by magento 2.

The second problem is that during the import with published products or which are published after during the import, data-definition slowed down after 5 products then takes hours to process the import.

Can you set up a fix to be compatible with the pimcore connector so that this type of loop no longer exists?

here a sample of my data-definition process :

4/4 [=] 100% (4 secs/4 secs) 38.2 MiB: Imported Object /Cegid/F00/F01/42E/42A02405          008002         X
Import finished!

 1/1 [=] 100% (< 1 sec/< 1 sec) 38.2 MiB: Imported Object /Cegid/D00/D01/41K/37DEB572                         X
Import finished!

 3/3 [=] 100% (2 secs/2 secs) 38.2 MiB: Imported Object /Cegid/C00/C10/32F/32FS00M03P                       X
Import finished!

 1/1 [=] 100% (< 1 sec/< 1 sec) 38.2 MiB: Imported Object /Cegid/C00/C12/34B/34B40185K                        X
Import finished!

 2/2 [=] 100% ( 1 sec/1 sec ) 38.2 MiB: Imported Object /Cegid/B00/B02/17C/17C04604BM           00A         X
Import finished!

 2/2 [=] 100% ( 1 sec/1 sec ) 38.2 MiB: Imported Object /Cegid/C00/C07/29E/29E512D                          X
Import finished!

Import finished!

 7/7 [=] 100% (32 secs/32 secs) 46.2 MiB: Imported Object /Cegid/C00/C10/32H/32HR00M0311SF                    X
Import finished!

 13/14 [>]  92% (7 mins/8 mins) 54.2 MiB: Import Object /Cegid/C00/C05/27C/27C24590R                        X

7 min for 13 products .......

here a sample of my magento bridge pimcore log

> [2020-04-24 14:20:08] StreamLogger.INFO: Pimcore data received:  {"product_id":69892,"store_view_id":0,"action":"insert/update"} []
> [2020-04-24 14:20:08] StreamLogger.INFO: 200 ["Product 69892 has been added to queue"] []
> [2020-04-24 14:20:08] StreamLogger.INFO: Pimcore data received:  {"product_id":69892,"store_view_id":0,"action":"insert/update"} []
> [2020-04-24 14:20:08] StreamLogger.INFO: 200 ["Product 69892 has been added to queue"] []
> [2020-04-24 14:20:08] StreamLogger.INFO: Pimcore data received:  {"product_id":69892,"store_view_id":0,"action":"insert/update"} []
> [2020-04-24 14:20:08] StreamLogger.INFO: 200 ["Product 69892 has been added to queue"] []
> [2020-04-24 14:20:09] StreamLogger.INFO: Pimcore data received:  {"product_id":69892,"store_view_id":0,"action":"insert/update"} []
> [2020-04-24 14:20:09] StreamLogger.INFO: 200 ["Product 69892 has been added to queue"] []
> [2020-04-24 14:20:09] StreamLogger.INFO: Pimcore data received:  {"product_id":69892,"store_view_id":0,"action":"insert/update"} []
> [2020-04-24 14:20:09] StreamLogger.INFO: 200 ["Product 69892 has been added to queue"] []

if i remove the publish attribut into my definition, there is no more problem and my import is fast.

thanks you very much for any help :)

Q	A
Bug report?	yes
Feature request?	no
Branch?	master

[dpfaffenbauer]

Magento Connector: Not a problem of data-defs. We cannot handle how other plugins integrate into Pimcore. You could try to disable event-listeners with DataDefs Events.

Performance: Also a Pimcore Problem, too much time already went into performance optimizing on data-defs side and pimcore side as well. nothing more we can do.

[The-Creeps]

In the first log below you can see the spam that your plugin sends through the pimcore connector and which slows down or completely freezes the import processing. When I simply import from the native pimcore system this does not happen and I only see one request per product .. I only do this when I use your plugin so it is linked ...

Magento LOG (import with w-vision)

[2020-04-24 14:20:08] StreamLogger.INFO: Pimcore data received:  {"product_id":69892,"store_view_id":0,"action":"insert/update"} []
> [2020-04-24 14:20:08] StreamLogger.INFO: 200 ["Product 69892 has been added to queue"] []
> [2020-04-24 14:20:08] StreamLogger.INFO: Pimcore data received:  {"product_id":69892,"store_view_id":0,"action":"insert/update"} []
> [2020-04-24 14:20:08] StreamLogger.INFO: 200 ["Product 69892 has been added to queue"] []
> [2020-04-24 14:20:08] StreamLogger.INFO: Pimcore data received:  {"product_id":69892,"store_view_id":0,"action":"insert/update"} []
> [2020-04-24 14:20:08] StreamLogger.INFO: 200 ["Product 69892 has been added to queue"] []
> [2020-04-24 14:20:09] StreamLogger.INFO: Pimcore data received:  {"product_id":69892,"store_view_id":0,"action":"insert/update"} []
> [2020-04-24 14:20:09] StreamLogger.INFO: 200 ["Product 69892 has been added to queue"] []
> [2020-04-24 14:20:09] StreamLogger.INFO: Pimcore data received:  {"product_id":69892,"store_view_id":0,"action":"insert/update"} []
> [2020-04-24 14:20:09] StreamLogger.INFO: 200 ["Product 69892 has been added to queue"] []

Magento LOG (Import with Native Pimcore System)

QUERY: instanceUrl=http://dev.magento.com&apikey=eb9402b0f11f67e4ce791405c32d0e186e9a33f193acbdd8c2d97fa18ea5024b&storeViewId=0
POST: message=product with ID 81860 successfully added to queue&status=SUCCESS&id=81860 [] []
[2020-04-27 10:29:14] StreamLogger.INFO: {"success":true,"msg":null} [] []
[2020-04-27 10:29:14] StreamLogger.INFO: 200 ["Product 81860 has been added to queue"] []
[2020-04-27 10:29:15] StreamLogger.INFO: Pimcore data received:  {"product_id":81861,"store_view_id":0,"action":"delete"} []
[2020-04-27 10:29:15] StreamLogger.INFO:
====REQUEST====
POST http://dev.pimcore.com/webservice/rest/product/update-status HTTP/1.1
Authorization: Bearer eb9402b0f11f67e4ce791405c32d0e186e9a33f193acbdd8c2d97fa18ea5024b
Content-Type: application/x-www-form-urlencoded
Accept: application/x-www-form-urlencoded

QUERY: instanceUrl=http://dev.magento.com&apikey=eb9402b0f11f67e4ce791405c32d0e186e9a33f193acbdd8c2d97fa18ea5024b&storeViewId=0
POST: message=product with ID 81861 successfully added to queue&status=SUCCESS&id=81861 [] []
[2020-04-27 10:29:15] StreamLogger.INFO: {"success":true,"msg":null} [] []
[2020-04-27 10:29:15] StreamLogger.INFO: 200 ["Product 81861 has been added to queue"] []
[2020-04-27 10:29:16] StreamLogger.INFO: Pimcore data received:  {"product_id":81862,"store_view_id":0,"action":"delete"} []
[2020-04-27 10:29:16] StreamLogger.INFO:
====REQUEST====
POST http://dev.pimcore.com/webservice/rest/product/update-status HTTP/1.1
Authorization: Bearer eb9402b0f11f67e4ce791405c32d0e186e9a33f193acbdd8c2d97fa18ea5024b
Content-Type: application/x-www-form-urlencoded
Accept: application/x-www-form-urlencoded

QUERY: instanceUrl=http://dev.magento.com&apikey=eb9402b0f11f67e4ce791405c32d0e186e9a33f193acbdd8c2d97fa18ea5024b&storeViewId=0
POST: message=product with ID 81862 successfully added to queue&status=SUCCESS&id=81862 [] []
[2020-04-27 10:29:16] StreamLogger.INFO: {"success":true,"msg":null} [] []
[2020-04-27 10:29:16] StreamLogger.INFO: 200 ["Product 81862 has been added to queue"] []
[2020-04-27 10:29:17] StreamLogger.INFO: Pimcore data received:  {"product_id":81863,"store_view_id":0,"action":"delete"} []
[2020-04-27 10:29:17] StreamLogger.INFO:
====REQUEST====
POST http://dev.pimcore.com/webservice/rest/product/update-status HTTP/1.1
Authorization: Bearer eb9402b0f11f67e4ce791405c32d0e186e9a33f193acbdd8c2d97fa18ea5024b
Content-Type: application/x-www-form-urlencoded
Accept: application/x-www-form-urlencoded

QUERY: instanceUrl=http://dev.magento.com&apikey=eb9402b0f11f67e4ce791405c32d0e186e9a33f193acbdd8c2d97fa18ea5024b&storeViewId=0
POST: message=product with ID 81863 successfully added to queue&status=SUCCESS&id=81863 [] []
[2020-04-27 10:29:17] StreamLogger.INFO: {"success":true,"msg":null} [] []
[2020-04-27 10:29:17] StreamLogger.INFO: 200 ["Product 81863 has been added to queue"] []

[dpfaffenbauer]

Yes, but this plugin doesn't do any special. It loads the import-source, loops through it, uses Pimcore Setters to set values, saves the object and done.

[The-Creeps]

Magento Connector: Not a problem of data-defs. We cannot handle how other plugins integrate into Pimcore. You could try to disable event-listeners with DataDefs Events.

Performance: Also a Pimcore Problem, too much time already went into performance optimizing on data-defs side and pimcore side as well. nothing more we can do.

Can you explain me how i can disable event-listener ? thx

[The-Creeps]

I suppose it's a no ? :)

[dpfaffenbauer]

sorry no. I am pretty certain it's not a problem caused by data-defs

[The-Creeps]

good answer ever on github .... thank you