search

teamtnt / laravel-scout-tntsearch-driver

Driver for Laravel Scout search package based on https://github.com/teamtnt/tntsearch
MIT License
1.1k stars 144 forks source link

Batch Import #33

Closed Melcus closed 8 years ago

Melcus commented 8 years ago

I want to import a table with about 5 million rows and it runs out of memory. Is there a way to import them in batches using the tntsearch:import?

Also, having filtering would make this package a must in every project.

nticaric commented 8 years ago

If you're using MySQL try to update TNTSearch to the latest version composer require teamtnt/tntsearch and then run tntsearch:import Model again.

Let us know if it works

Melcus commented 8 years ago

It's importing 1000 at a time, Well done ! Any news on filtering?

nticaric commented 8 years ago

Filtering is planned, but it's hard to say when exactly it'll be done since we're working on this project in our spare time

Melcus commented 8 years ago

Alright, thank you.

nticaric commented 8 years ago

@Melcus How long did it take for you to index those 5 million rows? And whats the search execution time?

Melcus commented 8 years ago

It goes at about 1000 / sec ( I'm on homestead, so theoretically, it can go even faster ). I didn't import all of them yet, the data is still growing. Only thing I tested was on 20k rows, ~54ms

nivanmorgan commented 7 years ago

If I'm trying to use it to create a search engine for a dynamic website with records updated daily, do I have to import the new records every time there is a new entry? Or can I import the Model once when I install it?

nticaric commented 7 years ago

@nivanmorgan you only import all of the records once, and thats it. Everything else is automatically handled by scout meaning it will sync the index on each, update, delete or insert.

prograamer commented 5 years ago

Hi,

Is there any way to disable handlers ?

I need to disable index at import data time and after that refresh index.

Automatic index synchronization make my data imports very slow.