Open andreas-gruenwald opened 4 years ago
might be challenging due to
so loading the dataobject will be necessary in any case, and most probably this is the most expensive operation (not extracting data from it for index)
might be challenging ...
inIndex()
and add rows to the store table regardless.ProcessPreparationQueue
will then load the entire data object and kick the irrelevant rows based on inIndex()
. Probably also the sub-IDs will then be created on the-fly.So the effort of loading data objects is the same, but it is shifted from the bootstrap command to the ProcessPreparationQueue, where it is easier to setup.
Maybe there is something I do not see yet.
but for knowing what IDs should be in index, you need to load the data object ... and the whole point of bootstrapping is knowing what IDs should be in index. processing preparation queue command already needs to know what IDs are in index.
or am I missing something?
Here is the difference of the two approaches. Let's assume that we have a system with 600.000 products. 200.000 are relevant for the product index.
BootstrapCommand
:
For 600.000 products, without parallelization, let's say that the command will run for 48 hours in a project where the product data model is complex.
BootstrapCommand
and ProcessPreparationQueue
:BootstrapCommand
:
Because the data objects are not loaded in this step, the process will probably terminate after a couple of minutes instead of hours/days.
ProcessPreparationQueue
:
BootstrapCommand
in step 1 will be scanned, as in_preparation_queue=1
(cost: low).There are two main differences:
BootstrapCommand
does not have the capability to restart, if the command is stopped unintentionally. Let's assume that the whole processing takes 48 hours, but after 30 hours on the DEV server the Symfony container is built. The command will stop, and restarting the BootstrapCommand
will result in another 48 hours run. With approach number 2 this won't happen, as the ProcessPreparationQueue
will only process the "open" records, not all data objects based on the product list condition. Mode number one is probably still needed, so those modes could coexist.We should definitly create something similar to what @andreas-gruenwald suggested. Currently it's really a very big challenge to do the bootstrapping in projects with many products and tenants.
are there any BC breaks needed for that?
are there any BC breaks needed for that?
Don't think that BCs are needed. Also, the current behavior could remain as the default one. The best way would probably be to implement a project related pull request.
I would love to see a PR for it :-)
Feature Request
I just have another idea in mind. Actually we could add a mode to the bootstrap command, so that it will only create "empty" ID-rows for non-existing ones, instead of processing the whole object(s):
https://github.com/pimcore/pimcore/blob/d4bf70250202c56fb81ece2d80a85c282daf67ac/bundles/EcommerceFrameworkBundle/Command/IndexService/BootstrapCommand.php#L131
The store table would then be used as a queue, and the ProcessPreparationQueue command would then take care about the rest:
in_preparation
is not set, the entries will be processed.This would depend on pimcore/pimcore#6487 though.