Incorrect Total Count in Laravel Scout Pagination

[Evenerik]

Scout Version

10.8.6

Scout Driver

Typesense

Laravel Version

10.48.3

PHP Version

8.3.0

Database Driver & Version

MySQL 8.0.36

SDK Version

No response

Meilisearch CLI Version

No response

Description

This is pretty much a reopening of @razvaniacob's issue, who has encountered the same problem I am currently facing. I won't go into too much detail since he described the bug perfectly well in his issue, but Scout's LengthAwarePaginator shows wrong total when using Typesense's engine

Steps To Reproduce

See steps in https://github.com/laravel/scout/issues/819

[driesvints]

@karakhanyans do you have any thoughts here?

[karakhanyans]

@driesvints will check this and get back with more info 👌

[razvaniacob]

I will be happy to see this solved guys. Good luck!

[driesvints]

@karakhanyans any news here?

[AbdullahFaqeir]

I'll be working on this

[tamakoma1129]

Hello,

I encountered the same issue reported here. In my case, I was using a model with a ULID as the primary key but initially forgot to set protected $keyType = 'string'; in my model. After updating this setting, I can confirm that the pagination now works correctly.

Here are the specifics of my development environment:

• Laravel Version: 10.48.4
• Scout Version: 10.8.6
• PHP Version: 8.1.27
• Scout Driver: Meilisearch
• Database: MySQL 8.0.36

Apologies if this solution has already been mentioned. I hope this helps someone else experiencing the same problem.

Thank you.

[driesvints]

Ping @jasonbosco

[jasonbosco]

@AbdullahFaqeir will be working on this going forward.

[AbdullahFaqeir]

Hi @Evenerik!

Can you please dump the collection schema, and sample record in typesense and the same record in database.

Below you can see a dump of my env showing a working fresh sample, I've seeded 10K users.

User model settings.

Search performed.

Paginated search results.

[driesvints]

Closing this issue because it's inactive, already solved, old or not relevant anymore. Feel to open up a new issue if you're still experiencing this.

[razvaniacob]

Is it possible to reopen this because the problem persists. I tracked it down to the method getTotalCount on Builder class https://github.com/laravel/scout/blob/10.x/src/Builder.php#L486

the $totalCount is correct but because of $this->queryCallback not being null, the total returned is the perpage number.

I don't really understand what is happening after that if statment `if (is_null($this->queryCallback)) { return $totalCount; } @driesvints Do I need to open a new issue about this?

[jasonbosco]

Could you share the information that @AbdullahFaqeir asked for here to debug this further: https://github.com/laravel/scout/issues/824#issuecomment-2131359416

[razvaniacob]

I'm happy to share more information. I've notice that things stop working as they should when using a callback query indicated in the documentation here: https://laravel.com/docs/11.x/scout#customizing-the-eloquent-results-query

The ecpected number of results should be 512 but they are limited to the number per page.

City model settings

Search performed.

Route::get('test', function () {
    return City::search('buzau')
        ->query(fn (Builder $query) => $query->with(['county']))
        ->paginate(10)->onEachSide(1)->withQueryString()
        ->through(fn ($obj) => [
            'name' => $obj->name,
            'city' => $obj->county?->name ?? '',
        ]);
});

Paginated search results.

{
  "current_page": 1,
  "data": [
    {
      "name": "Buzau",
      "city": "Judetul Buzau"
    },
    {
      "name": "Comuna Munteni Buzau",
      "city": "Judetul Ialomita"
    },
    {
      "name": "Dealul Viei",
      "city": "Judetul Buzau"
    },
    {
      "name": "Dealul Viei",
      "city": "Judetul Buzau"
    },
    {
      "name": "Valcele",
      "city": "Judetul Buzau"
    },
    {
      "name": "Trestioara",
      "city": "Judetul Buzau"
    },
    {
      "name": "Suditi",
      "city": "Judetul Buzau"
    },
    {
      "name": "Satu Nou",
      "city": "Judetul Buzau"
    },
    {
      "name": "Recea",
      "city": "Judetul Buzau"
    },
    {
      "name": "Posta",
      "city": "Judetul Buzau"
    }
  ],
  "first_page_url": "https://bliss-spa.test/test?query=buzau&page=1",
  "from": 1,
  "last_page": 1,
  "last_page_url": "https://bliss-spa.test/test?query=buzau&page=1",
  "links": [
    {
      "url": null,
      "label": "« Previous",
      "active": false
    },
    {
      "url": "https://bliss-spa.test/test?query=buzau&page=1",
      "label": "1",
      "active": true
    },
    {
      "url": null,
      "label": "Next »",
      "active": false
    }
  ],
  "next_page_url": null,
  "path": "https://bliss-spa.test/test",
  "per_page": 10,
  "prev_page_url": null,
  "to": 10,
  "total": 10
}

I verified the total number in the Typesense Dashboard

So I think the problem lies somewhere in the getTotalCount method in the Builder class https://github.com/laravel/scout/blob/10.x/src/Builder.php#L486

Please let me know how else I can help with this.

Thank you!

Raz

[AbdullahFaqeir]

Hey @jasonbosco @razvaniacob

I don't know how I've seen the whole execution within my head to figure this out 😅 I literally didn't even go through the code.

What's actually causing the issue is what happens when you apply a customisation to the query (db) which happens after the (search) query.

When @razvaniacob indexed his data, he didn't index the (id) of the city, that's first.

After the search results is returned normally with the correct total number count, while not just typesense, even other engines, generate a list of the plucked id's from the (search) query, using this list of id's the fetch the actual entities inside the database on order to apply the (db) query customisation on them.

In this case, there was no id's being returned from the (search) query, thus, when trying to execute the (db) query customisation, it returns null because, you know! 😅.

So I believe the only way for this to be fixed is to force index the models id's.

[razvaniacob]

Good observation. But actually the id is indexed to typesense and you can see the id in the typesense dashboard screenshot, but even with that I get the same result.

This is what I have in my City model:

[AbdullahFaqeir]

Good observation. But actually the id is indexed to typesense and you can see the id in the typesense dashboard screenshot, but even with that I get the same result.

This is what I have in my City model:

Tho is it defined in the typesense scheme?, check your previous comment

[razvaniacob]

I added the id to the typesense scheme and reindexed and still no change in the results. I get 10 results (the per page total) instead of 512

[razvaniacob]

@AbdullahFaqeir do you need anything else from me in order to finally find a fix for this problem?

[jasonbosco]

Mostly for @AbdullahFaqeir: Quick note that the id field is automatically generated by Typesense if a document sent to Typesense does not have a field called id defined in the document (even if the id field is not defined in the collection schema).

But if an id field is explicitly set in the document, then the specified value is used as the id of the document.

Unless exclude_fields or include_fields search parameters are used to exclude the id field from the Typesense search API response, this field will always be returned by Typesense.

[jackkitley]

Have the same issue when using ->query()

Issue is over 2 years old now.

        return $this->model->search($search)
            ->query(fn(Builder $query) => $query->with(['mainListingImage']))
            ->options([
                'query_by' => 'vehicle_description, transmission, fuel_type, colour, year'
            ])
            ->paginate()
            ->withQueryString();

[tharropoulos]

After spending some time debugging, I've come to the following findings:

When using the paginate function in Laravel Scout with the Typesense engine, the number of records returned is always limited to the perPage value, regardless of the actual total number of matches on the Typesense server. This behavior stems from the interaction between various methods within the Builder and TypesenseEngine classes.

Details

1. Hits on the Current Page:
   The line here https://github.com/laravel/scout/blob/991056204a410fc7a9c22b1ec9bbbae6bb675241/src/Builder.php#L496 returns the number of hits on the current page, which will always be capped by the limit (default is Eloquent's perPage 15, I go into more detail about this below). This is evident here https://github.com/laravel/scout/blob/991056204a410fc7a9c22b1ec9bbbae6bb675241/src/Engines/TypesenseEngine.php#L357-L362 As mapIdsFrom just returns the mapIds function: https://github.com/laravel/scout/blob/991056204a410fc7a9c22b1ec9bbbae6bb675241/src/Engines/Engine.php#L111-L114

2. Mutation of the ids Variable:
   The ids variable, as a result, is always less than the total matches, and it is then mutated by the keys method, which uses the tap function to return a cloned builder object: https://github.com/laravel/scout/blob/991056204a410fc7a9c22b1ec9bbbae6bb675241/src/Engines/Engine.php#L122-L125 This, in turn, will just return the same number as the previous call to the mapIds function.

3. Limit Setting Logic:
   The logic here: https://github.com/laravel/scout/blob/991056204a410fc7a9c22b1ec9bbbae6bb675241/src/Builder.php#L500-L502 attempts to set the limit, but since the limit is initialized and always less than the totalAmount variable, this condition never triggers. Thus, the number of total records in TypesenseEngine defaults to the perPage value.

4. Why: The paginate function on TypesenseEngine https://github.com/laravel/scout/blob/991056204a410fc7a9c22b1ec9bbbae6bb675241/src/Engines/TypesenseEngine.php#L206-L214 Accesses the builder's take function, which sets the limit attribute to the perPage parameter, which is never null. This way, the aforementioned check will always result in what perPage is equal to, and in turn, will result in the behavior you encountered.

Fix

Remove the assignment of perPage's value to the builder's limit attribute. I'll create a relevant PR shortly, so you can check it out yourselves.

[razvaniacob]

Wow.. good job @tharropoulos I was able to track down the issue you found but didn't have time to come up with a fix. I wasn't sure if removing the assignment of perPage to the builder's limit attribute won't affect other search engines, because I found it odd that it was there to begin with.

I'm happy you're on this problem!

[tharropoulos]

Wow.. good job @tharropoulos I was able to track down the issue you found but didn't have time to come up with a fix. I wasn't sure if removing the assignment of perPage to the builder's limit attribute won't affect other search engines, because I found it odd that it was there to begin with.

I'm happy you're on this problem!

I've manually tested it but haven't created any new tests to verify. If need be, I'll be sure to include some! Rest of the tests seem to pass as expected