search

pimcore / data-importer

This extension adds a comprehensive import functionality to Pimcore Datahub.
Other
40 stars 60 forks source link

[Bug]: The "kernel" service is synthetic error when processing parallel #420

Open IronSean opened 3 months ago

IronSean commented 3 months ago

Expected behavior

datahub:data-importer:process-queue-parallel will properly process rows, or throw errors when there is a logic error

Actual behavior

Occasionally rows fail with the following error: Error processing element: IMPORT,DATA,HERE The "kernel" service is synthetic, it needs to be set at boot time before it can be used.

This seems like a possible race condition as it seems to happen more when the server is under load by other tasks, but I might be imagining that.

Steps to reproduce

Inconsistent. Run lots of 10k, 100k, 1m row datahub imports with multiple parallel workers (8 in my case on 4 cores) while the server is under other workloads.

Sometimes there is no error, sometimes a few rows, and sometimes the whole import fails.

github-actions[bot] commented 2 months ago

Thanks a lot for reporting the issue. We did not consider the issue as "Pimcore:Priority", "Pimcore:ToDo" or "Pimcore:Backlog", so we're not going to work on that anytime soon. Please create a pull request to fix the issue if this is a bug report. We'll then review it as quickly as possible. If you're interested in contributing a feature, please contact us first here before creating a pull request. We'll then decide whether we'd accept it or not. Thanks for your understanding.

abdelrhmanyosry commented 2 months ago

I have the same issue. image

stephangmms commented 3 weeks ago

Hi everyone,

is there something new in this? I also occur this error since a few days?

fashxp commented 3 weeks ago

we are not able to reproduce it, see also thread here, which is a similar topic. any additional insights or debugging results are welcome.

abdelrhmanyosry commented 3 weeks ago

@stephangmms it's happening when importing a large number of products with a large number of attributes we had more than 200+ attributes for each product.

stephangmms commented 3 weeks ago

@stephangmms it's happening when importing a large number of products with a large number of attributes we had more than 200+ attributes for each product.

that sounds familiar. Sadly i have not too much insight into that problem because i was out of this project for a while. I only know that the data-importer jobs (csv-imports, parallel execution) run in small scale test (<100 items) perfectly. Now as we are testing 5000+ products with only 3 attributes/product this problems occur.

[DATA-IMPORTER] Error processing element: 0695, --- DATA REMOVED FOR PRIVACY REASONS --- , Symfony\Component\DependencyInjection\Exception\ServiceNotFoundException: The "kernel" service is synthetic, it needs to be set at boot time before it can be used. in /var/www/html/vendor/symfony/dependency-injection/Container.php:246

Pimcore: 11.3.0 DataHub: 1.7.2 Data-Importer: 1.9.0