search

akeneo-labs / EnhancedConnectorBundle

[DEPRECATED] Provides product reader with more option (choice on completeness, choice on enabled, delta based on last export time).
21 stars 11 forks source link

Product Export fails #15

Closed AlexanderHuyghebaert closed 8 years ago

AlexanderHuyghebaert commented 9 years ago

Hello,

We have a setup with 5000 products and for each product 1-10 images. When we do a full product export, the CSV is made, but when the zip of images is made, the export fails. It looks like it fails because there are too many images.

Is there a possibility to export in batches?

damien-carcel commented 8 years ago

Hello Alexander.

What error message do you have when trying to do the export? What version of the PIM are you using, and are you using MySQL or MongoDB?

Regards, Damien

AlexanderHuyghebaert commented 8 years ago

Hello,

There is no error message, it just keeps running and after a time we get a memory limit on the server. PIM version is: 1.3.21 Hare Force

MongoDB is used for Products data

Best regards, Alexander

damien-carcel commented 8 years ago

Hi Alexander.

How much memory do you set for the php command line ? For export like that, we recommend at least 768 Mo, but 1 Go would be wise. If you are in development mode and/or have XDEBUG activated, then you can push it to 2 Go !

Regards, Damien

AlexanderHuyghebaert commented 8 years ago

Hello,

The memory limit is now set on: 4096M

Best regards, Alexander

damien-carcel commented 8 years ago

So you can confirm me that the export works with a higher memory limit? If yes, I will close this ticket.

Regards, Damien

AlexanderHuyghebaert commented 8 years ago

Hello,

No, I have memory limit of 4GB and it still does not work...

Regards, Alexander

damien-carcel commented 8 years ago

Hello Alexander.

Just to be sure it is a problem with the connector, and not with the PIM, could you try to run an export with the standard product export ?

Regards, Damien

bhouillon commented 8 years ago

Hello everyone,

I have the same problem, I have a set of 11K products to export, I have allocate 4096M of memory. The export crashes around 4900 products.

Regards Benjamin

damien-carcel commented 8 years ago

Hello Bhouillon.

As I suggest to Alexander, is this problem specific to the Enhanced connector, or can you reproduce it with the standard CSV product export? If you're in the second case, then you should probably ask your questions on Akeneo forums, where more people will be able to answer.

What version of Akeneo are you using btw, with which storage?

Regards, Damien

bhouillon commented 8 years ago

capture d ecran de 2015-11-18 15 33 20 Hello Damien, I think it's more an Akeneo problem, because I have the same behavior with DnD MagentoConnectorBundle. Besides that, I have another problem with the Enhanced connector bundle, it continues to run even after finishing to export, it has status "Terminated" and status "Started" in Akeneo, so if we stop the task the status is "failed" and "terminated". Btw delta export will not export anything.

Regards, Benjamin

ghost commented 8 years ago

Hi !

Do you have news ?

I have the same problem described earlier by bhouillon. Did you find a clue on what's happening ? It's critical for me because the export never ends. I tried with standard export and it worked correctly. Moreover, I made a test setting the cli memory_limit to -1 (do not try this at home) and still the same problem.

Could you please check this ?

Thanks in advance.

Best regards.

damien-carcel commented 8 years ago

Hello @gaces, @bhouillon, @AlexanderHuyghebaert.

We recently fix a memory leak issue with the product writer, both for Akeneo 1.3 and 1.4. It is the same writer that is used by the Enhanced connector bundle and the standard Akeneo product export.

So updating to the last version of Akeneo 1.3 or 1.4 should then fix you problems. You can see the fix here for 1.4: https://github.com/akeneo/pim-community-dev/commit/656b1afd5b68fb91e5332447719bd143b9f151a7 (it has been done also in 1.3).

Regards, Damien

ghost commented 8 years ago

Hello,

Thanks for your answer, however @bhouillon and I are currently working on Akeneo 1.3.32, could you send me the diff related to the fix for this version ?

Thanks,

Regards.

damien-carcel commented 8 years ago

Here is the fix for 1.3: https://github.com/akeneo/pim-community-dev/commit/0592482562b76b78565b5ffb7989af286108b4e5

However, what issue are you referring to? The memory leak on export, or the problem with the job status? The fixes I refer too are for the memory leak. Please open a new issue for the status problem, or it will make this conversation hard to understand.

Regards Damien

ghost commented 8 years ago

Thanks for the link.

I was refering to the memory issue. I created a new issue (https://github.com/akeneo-labs/EnhancedConnectorBundle/issues/34) about the status of the exports.

Regards.

damien-carcel commented 8 years ago

Hello here. A memory leak issue has been fixed on PIM exports, which also affect this connector.

It has been fixed on both Akeneo 1.4.9 and 1.3.32 (you can see the PR on 1.4 here).

Please check if your problem is solved by updating to the last version of the PIM.

Regards, Damien

gplanchat commented 8 years ago

I have the same issue here.

I'm trying to export 4k products, with 13Gb of media assets, using command line php -d memory_limit=4G app/console akeneo:batch:job my_job

The problem seems to be specific for writer classes implementing Pim\Bundle\BaseConnectorBundle\Writer\File\ArchivableWriterInterface and the zip archive generation, not the Enhanced Connector.

I'm searching for a solution.

Greg

damien-carcel commented 8 years ago

Hello Greg.

So it does seems to be a PIM related problem. Don't hesitate to post you problem (and link it here) on our forum. You will have more chance to obtain a solution in here. However, I see in your command line that you are launching the export in dev mode. This will cause a memory leak (and it is perfectly normal). Try launching the command in prod mode:

php -d memory_limit=4G app/console --env=prod akeneo:batch:job my_job

What's more, you're talking about media assets. Does that mean you're using the Enterprise Edition of Akeneo? Because if you are, then you can raise this bug on our Service Desk platform as it affects the PIM itself, so it will be treated in priority by our support team.

Regards, Damien

gplanchat commented 8 years ago

Hello Damien,

No it's not EE, it's a custom media assets management in Akeneo.

Greg

damien-carcel commented 8 years ago

Hello Greg.

As it is custom development, there is not much we can do here. The only way we could help in such case is if your company takes support days from our service team. However, there was a memory leak on the product export that has been recently corrected. Be sure you’re using the last version of Akeneo, (again, in prod mode).

Last question: what writer are you using? Is it the Pim one or your own? Cause if you’re using the PIM one, you definitely need to report it on our forums, so our Core team can take this bug into account.

Regards, Damien

manh-nguyen commented 8 years ago

Hi @damien-carcel , I tried to export 22k products using this connector and it's always failed at read 2k-5k. I've try to use both MySQL and MongoDB for pim_catalog_product_storage_driver but they had the same error. I've also tried to export it using Akeneo Native Product Export CSV and it worked as well with both MySQL and MongoDB.

My local server is quite good: CentOS 7, PHP 5.5.30, 12GB of RAM, SSD storage, core i7, MEMORY_LIMIT = 2GB in php.ini.

Please see the error and try to check it:

If you want to have the init.xlsx + products.csv for testing, please tell me, I will share you the file via google drive.

Regards, Manh

damien-carcel commented 8 years ago

Hello Mahn.

There were indeed a memory leak. It should be fixed now with the last update of the bundle (1.0.5). @AlexanderHuyghebaert and @bhouillon, this should fixe your problem too.

Regards, Damien

manh-nguyen commented 8 years ago

Hi @damien-carcel ,

Thank you very much for this update. It's really much better than the old one. The old one always fail at reading 2k-5k records, now it can read more than 16k records. Sadly It will be failed after16k reading due to memory issue: http://i.imgur.com/wigUWQx.png

[2016-01-11 18:00:15] [pim_enhanced_connector.export.product.title] PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 32 bytes) in pim/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php on line 2722

[2016-01-11 18:43:08][pim_enhanced_connector.export.product.title] PHP Warning: Error while sending QUERY packet. PID=18488 in pim/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php on line 855 [Doctrine\ORM\ORMException] The EntityManager is closed.

[2016-01-12 09:30:23] PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 32 bytes) in pim/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php on line 2632

We have 22k products there.

Please have a look at Akeneo Native CSV export product. It can export 30k products or more as well.

Cheers, Manh

damien-carcel commented 8 years ago

Hello Manh.

I just remember there is a memory leak caused by a bug in the object detacher of the PIM. This has been fixed in Akeneo 1.4.15, and I see in your previous post you are in 1.4.14. I recommend you to upgrade to the last 1.4 version of the PIM and test if the memory leak is gone.

Regards, Damien

manh-nguyen commented 8 years ago

Hi @damien-carcel , I used composer to update the latest version of Enhanced Connector bundle and It also updated Akeneo to version 1.4.16 (Version (EE):1.4.16 SamaraVersion (CE):1.4.16 Rhapsody Rabbit ).

vendor/akeneo-labs/pim-enhanced-connector/Pim/Bundle/EnhancedConnectorBundle/CHANGELOG.md

1.0.5 (2016-01-11)

Bug fix:

http://i.imgur.com/E5aBU7H.png {quote} read 16530 2016-01-15 6:50:04 PM ENHANCED_EXPORT_PRODUCT_CSV An exception occured during the job execution PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 79 bytes) in pim/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php on line 2710 {quote}

Regards, Manh

fitn commented 8 years ago

Hi,

Thank you for raising this issue. We're currently working on this memory leak and we will release a new version when it will be fixed.

Regards,

fitn commented 8 years ago

Hi,

Here is a documentation to explain and fix this problem: http://docs.akeneo.com/1.4/reference/scalability_guide/more_than_1GB_of_product_media_to_export.html As it comes from the product itself and we need to avoid Backward Compatibility breaks, we provide a way to fix this issue directly on your projects.

Regards,