Non-eager updates

[Thinkscape]

Because of the nature of ElasticSearch, index updates should not be real-time with main-db updates for most cases, because that puts a write-load on the index database.

There are usually several ways to handle this (or combination of those):

• periodic updates
• bulk (buffer with size) updates
• stacked (timed buffer) updates

UC1: A wiki page with hundreds of write ops / minute on ORM entities. The index (used for searching) does not have to be updated after each small change, but each 5 or 10 minutes.

UC2: User database - permission updates need to be updated on the spot, however user information (i.e. name, address, description, profile) can be synced every few minutes or each X flushes.

[Ocramius]

@Thinkscape bulk operations shouldn't probably go through PHP anyways in my opinion... hydration would be too slow for bulk operations.

As for the writes, we could just push them to a queue instead of directly asking ES.

That would probably be simpler

[macnibblet]

For bulk operations: The proposed solution would be to implement a amqp and let the the user decide what ever messaging system he wants.

We should also add a property prioritisation with three different levels

1, IGNORE - not related to ES skip bulk 2, BULK (default) - should be placed in the amqp if it setup else sync with es 3, IMMIDIATE - sync directly with ES

[Ocramius]

As discussed on IRC, I think a simple "write" backend adapter would solve the problem and let anyone plug in any kind of logic.

It would also ease operations for mocking/testing.

That allows for various implementations:

• writing to /dev/null (seriously, only for testing :-)
• writing directly to ES as soon as data is flushed to the DB
• collecting data from a flush, package it and send it bulk to ES
• writing to a different adapter (a message queue or whatever) for later pickup