search

Pylons / substanced

An application server based on the Pyramid web framework (http://substanced.net)
Other
156 stars 58 forks source link

Indexes should probably not assume MODE_ATCOMMIT if queue processor is not active #61

Open mcdonc opened 11 years ago

mcdonc commented 11 years ago

Right now, if the queue processor is inactive, indexes will presume that they should not add to the queue (e.g. MODE_DEFERRED), and instead, they should treat indexing operations as if they were MODE_ATCOMMIT. This foils bulk load scenarios where you don't actually want to index the data while the loading is happening.

However, right now, the queue is one bigass nonpersistent object. This means that if it's not popped every so often, transactions which append to it need to write a multimegabyte list on every commit.

I'm not sure how to solve this yet.

mcdonc commented 11 years ago

Portential solution: have each transaction create its own list of persistent actions and manage the lists in the _p_resolveConflict of a persistent object that keeps a set of them.

mcdonc commented 11 years ago

Note that the bulk loading scenario is now handled by the substanced.catalogs.force_deferred flag (https://github.com/Pylons/substanced/commit/adecc33f67aa053e740ed983bb6fe1285f751d93). But the issue still remains of reserializing the bigass nonpersistent queue on every commit that adds to it.