search

pimcore / data-importer

This extension adds a comprehensive import functionality to Pimcore Datahub.
Other
38 stars 56 forks source link

[Improvement]: big imports fails due to timeouts #386

Closed nosovk closed 8 months ago

nosovk commented 8 months ago

Improvement description

When we're running a big import job. Like 300'000 new products. And it fails because of timeouts. We could fix that by adding set_time_limit into worker code. In that case

kingjia90 commented 8 months ago

Thank you for reporting, please provide a PR for it with the exact fixes so that we can review and merge it

SamyMP commented 8 months ago

Hi @nosovk, are your timeouts linked to MySQL ? We are running into similar issues lately in one of our environments, where we get MySQL has gone away or SQLSTATE[HY000] [2002] Connection refused when trying to import a lot of dataobjects, we are not sure whether it is due to the server not having enough memory or because of a recent release of the dataimporter

nosovk commented 8 months ago

Hm, nope, it's not SQL error, it was from PHP. But I'll check resource load during the import

nosovk commented 8 months ago

nope, there is even no spike in resource consumption :(

fashxp commented 8 months ago

It seems that you run the command-based execution of the import. That command always runs all the items from the queue - anf if there are a lot, it takes time ... and might hit the execution limit of your php processes. There is not much we can do about that as this is connected to your infrastructure configuration.

Switching to symfony messenger based execution might solve that issue though, see also https://pimcore.com/docs/platform/Data_Importer/Installation#symfony-messenger-based