search

manticoresoftware / manticoresearch-php

Official PHP client for Manticore Search
MIT License
166 stars 32 forks source link

Method to upload a large batch at max speed #217

Open marclaporte opened 1 week ago

marclaporte commented 1 week ago

Could / should the Manticore PHP client be adapted?

2024-09-23_170943

sanikolaev commented 1 week ago

Hi @marclaporte

  1. disable binlog

This has been mitigated here https://github.com/manticoresoftware/manticoresearch/issues/879

  1. write to multiple tables at once

This will be resolved once auto-sharding is completed:

Also, I believe the client shouldn't manage the schema directly anyway.

  1. write with a higher concurrency
  2. write with a larger batch

It might make sense to integrate this into the client. The idea would be that when the client receives a large batch of documents, it starts sending them to Manticore at the maximum possible speed, automatically adjusting concurrency and batch size. This could be based on factors like the number of CPU cores and workers available in the Manticore instance, increasing until further gains are no longer feasible.