Scout import for large datasets fails after upgrading from 10.X to Laravel 11.X

[pritborteam]

Scout Version

^10.8

Scout Driver

Typesense

Laravel Version

11.4.0

PHP Version

8.3

Database Driver & Version

MariaDB-10.11.7

SDK Version

0.26

Meilisearch CLI Version

No response

Description

I had no issue with import/flush with latest laravel Scout with Laravel 10.X and using typesense v.0.26. Post migration to Laravel 11.X, I am unable to import large data using scout:import command for typesense. After indexing first 500 documents, it says my collection with the index name already exists. Issue is not visible with small data less than 500.

Steps To Reproduce

1. Create a table with large data. May be 5000 or 10000.
2. Do follow the steps of creating toSearchableArray() and Collection as per documentation.
3. Use "php artisan scout:import "App\Models\YourModel"
4. After indexing first 500, you will see error saying collection with the name already exists.
5. On typesense server I find first batch of initial 500 records already indexed.
6. So while attempting next batch to index, it says collection is already present.

[jasonbosco]

Could you try setting connection_timeout_seconds in the typesense section in scout config file to say 900?

[pritborteam]

Could you try setting connection_timeout_seconds in the typesense section in scout config file to say 900?

This does not solve the problem. But when try cleaning using php artisan cache:clear This does the job well.

But I have below scenarios for issue and its very evident it is something to do with cache:-

Scenario: 1

1. Scout Flush
2. Scout Import (Issue is seen)

Scenario: 2

1. Scout Flush
2. Scout Import (Issue is seen)
3. Clear cache using artisan command
4. Then Scout Import again (Issue is not seen) and import is successful.

Scenario: 3

1. Scout Flush
2. Clear cache
3. Scout Import (Issue seen)

Scenario: 4

1. Clear cache
2. Scout Flush
3. Scout Import (Issue seen)

Observation: The issue will be always visible when you run the Scout Import for the first time. The only way to solve it is to first run the Scout Import once, let it fail and then run artisan clear cache and then run Scout Import again. This is not great.

BTW i use Redis for cache. But I was using this even in Laravel10 and had no issue.

[driesvints]

Hi @pritborteam. Looks like you found a solution to your issue so I'm going to close this one. I don't feel we need to take action right now because we only have one report for this, sorry.

[pritborteam]

Hi @pritborteam. Looks like you found a solution to your issue so I'm going to close this one. I don't feel we need to take action right now because we only have one report for this, sorry.

But this issue is evident. Why do I need to clear the cache only after every first failure? Clearing the cache before even issue is seen is not working.

[pritborteam]

This issue is also seen when you completely clear all database and run seeder again.

[AbdullahFaqeir]

I'll take a loot into this

[erichowey]

I'm also seeing this issue when my database seeders run. There's no issue when I try to import:

[pritborteam]

I'm also seeing this issue when my database seeders run. There's no issue when I try to import:

To see the issue in Scout Import, try having dataset more than 500 if your batch import size of 500. if fails in second iterationa. In seed, i guess it is one file each but in Scout Import it is batch import. Resolution is only after first failure cache clear.

[voveson]

I'll take a loot into this

Hi Abdullah! Just wondering – have you had a chance to look into this yet?