Out of memory on feed generation

[klau85]

When i run the command to generate the feed _php -d memorylimit=8G bin/console tweakwise:generate-feed -q i get the error

PHP Fatal error:  Allowed memory size of 8589934592 bytes exhausted (tried to allocate 20480 bytes) in vendor/shopware/core/Framework/DataAbstractionLayer/FieldSerializer/PHPUnserializeFieldSerializer.php on line 36
PHP Fatal error:  Allowed memory size of 8589934592 bytes exhausted (tried to allocate 65536 bytes) in vendor/symfony/error-handler/DebugClassLoader.php on line 300

I only have 500 products in my database and the generate feed command fails even with 8G of memory

Need to mention that i have 16GB of RAM on my server (10GB free) OS Debian buster php 8.2.21 shopware 6.6.1.0 Tweakwise plugin 3.0.0

[klau85]

Can someone take a look, please?

[haassie]

Hi Clause. Sorry for the late response. Holiday season...

Just to get a better idea of your environment: you say you have 500 products, but how many properties are attached to these products?

[klau85]

The content of feed is the default one, i didn't change anything

[klau85]

Also, i customized the feed as described here https://github.com/richardhaeser-com/sw-tweakwise/blob/main/DOC.md#customize-feed but that doesn't fix things

The problem seems to be with the query from FeedService, line 179. It chooses all the product info, no matter what fields i require in my template.

If i remove from the query the associations i don't need (Eg.: options, children), then the feed generator command works as expected.

Is there a way out of the box to configure what fields i need for my feed ? So the query doesn't get bloated with extra stuff.

[haassie]

Hi, it is not about the template indeed but about the data attached to the products. At the moment it is not possible to select the fields, but this might be possible in an upcoming version.

BTW: we tested this on quite some different datasets and this is the first where this is happening. I guess, a lot of options and variants are used in your setup? Do you have some numbers for me so I can try to reproduce the issue?

[klau85]

i have 574 active products but in the products table are 10,910 rows in product_option table i have 44,274 rows in property_group_option i have 10,533 rows

[haassie]

Those amounts should not be causing a lot of issues. I'm currently working on an update where the data that needs to go in the feed can be set in the admin. I expect this update today / tomorrow.

[haassie]

Hi Clause, are you able to test the latest version from the main branche? Some options are added to the feed settings (in the administration). Please let me know if this resolves your issues.

[klau85]

Hi, I just installed store.shopware.com/rhaetweakwise": "3.0.2", and on "Tweakwise Feed" section the only new setting i see is the "Generate now" button. After i click on it, it doesn't actually generate the feed, it just leaves the status "Will be regenerated on Within one minute"

[haassie]

I would suggest to use the version from packagist and use the dev-main branch. So composer require richardhaeser/sw-tweakwise:dev-main. You should have some options in the bottom of the screen after that.

[klau85]

i installed richardhaeser/sw-tweakwise:dev-main and now i see the options for Data and new settings in General When i click on "generate now", no feed is generated. the same if i run the command php -d memory_limit=8G bin/console tweakwise:generate-feed -q

[haassie]

The command is now using the message queue. Do you have that running or is it running from the admin?

[klau85]

we are not using the message queue. What do you mean by "is it running from the admin" ?

[haassie]

Is there a reason why u are not using the message queue? I highly recommend to use it. See https://developer.shopware.com/docs/guides/hosting/infrastructure/message-queue.html. There is also more information on the admin worker.

[klau85]

Hi,

I run the command bin/console messenger:consume failed async low_priority scheduler_shopware

i clicked on "Generate now" from "Tweakwise Feed" section but nothing happens.

Feed configuration:

Am i missing something? What should i do to generate the feed ?

[haassie]

Is it possible to get access to a dev/test environment so I can check and help? If so, please send me an email on richard@richardhaeser.com

The message consumer should create a file on the path that is mentioned in the top of the feed configuration screen.

[klau85]

yes, i'll get back with the details on email

[haassie]

As discussed outside Github: the code within the main branch should solve the problem now. Please let me know if this solved your problem.