FriendsOfSymfony / FOSElasticaBundle

Elasticsearch PHP integration for your Symfony project using Elastica.
http://friendsofsymfony.github.io
MIT License
1.24k stars 795 forks source link

Populate command does not loop through every elements #1184

Closed ragusa87 closed 6 years ago

ragusa87 commented 7 years ago

When I launch a populate command (with reset), the progress bar does not complete to 100% on some types. There is no error, bulk is processed as expected by elasticsearch. (I added an echo output for debug process)

php app/console fos:elastica:populate --no-debug

image

https://github.com/FriendsOfSymfony/FOSElasticaBundle/blob/3.2.1/Doctrine/AbstractProvider.php#L89

it's seems that the $offset is incremented by $options['batch_size'] instead of $sliceSize. Using $sliceSize seems to fix the problem.

The query used to index elements is here:

        $qb = $this->createQueryBuilder("e")
            ->select("e,d")
            ->join("e.dates", "d")
            ->join("d.placeArea", "pa")
            ->join("e.organisation", "o")
            ->where("d.endDate > :ago or d.endDate is null")
            ->setParameter("ago", new \DateTime("2 months ago"))
            ->andWhere("e.status in (:status)")
            ->andWhere("o.status in (:status)")
            ->setParameter("status", [StatusInterface::STATUS_APPROVED, StatusInterface::STATUS_MODIFIED]);

I am using with php5.6+, vendors are below:

composer show | grep -E "(elastic|doctrine)"
# doctrine/annotations                    v1.3.0             Docblock Annotations Parser
# doctrine/cache                          v1.6.1             Caching library offering an object-oriented API for many cache backends
# doctrine/collections                    v1.3.0             Collections Abstraction library
# doctrine/common                         v2.6.2             Common Library for Doctrine projects
# doctrine/data-fixtures                  v1.2.2             Data Fixtures for all Doctrine Object Managers
# doctrine/dbal                           v2.5.5             Database Abstraction Layer
# doctrine/doctrine-bundle                1.6.4              Symfony DoctrineBundle
# doctrine/doctrine-cache-bundle          1.3.0              Symfony Bundle for Doctrine Cache
# doctrine/doctrine-fixtures-bundle       2.3.0              Symfony DoctrineFixturesBundle
# doctrine/doctrine-migrations-bundle     v1.2.1             Symfony DoctrineMigrationsBundle
# doctrine/inflector                      v1.1.0             Common String Manipulations with regard to casing and singular/plural rules.
# doctrine/instantiator                   1.0.5              A small, lightweight utility to instantiate objects in PHP without invoking their constructors
# doctrine/lexer                          v1.0.1             Base library for a lexer that can be used in Top-Down, Recursive Descent Parsers.
# doctrine/migrations                     1.4.1              Database Schema migrations using Doctrine DBAL
# doctrine/orm                            v2.5.6             Object-Relational-Mapper for PHP
# friendsofsymfony/elastica-bundle        v3.2.1             Elasticsearch PHP integration for your Symfony2 project using Elastica
# gedmo/doctrine-extensions               v2.4.25            Doctrine2 behavioral extensions
# ruflin/elastica                         2.3.1              Elasticsearch Client
# stof/doctrine-extensions-bundle         dev-master 4e7499d Integration of the gedmo/doctrine-extensions with Symfony2

Can you please take a look at this ?

makasim commented 6 years ago

The provider API you are referring to was deprecated and removed in v5 with a completely new solution based on pagerfanta. Should work in v5 version.