Bulk processing

[mvrhov]

What's the best way to process something in bulk when using iterator e.g.

$foos = $model->findAll();

foreach ($foos as $foo) {
    if (foo) {
        $existed++;
    } else {
        $model->deleteOne($foo);
        $removed++;
    }

    unset($foo);
}

unset doesn't seem to do anything. And the memory just keeps going up.

[chanmix51]

That’s because of the IdentityMapper. It keeps a cache of all the instances fetched from the database to ensure that if a record is fetched twice it will return the same instance hence an update on the instance will update it everywhere.

The model manager is an entity CRUD oriented model. What you are trying to do in PHP would by far be better done in SQL using the SimpleQueryManager:

WITH
   deleted_foo AS (DELETE FROM foo WHERE {condition} RETURNING foo_id)
SELECT 
  count(foo.foo_id)         AS existing_count,
  count(deleted_foo.foo_id) AS deleted_count
FROM foo, deleted_foo

[mvrhov]

Well that was not an good example :) What to remove was to be decided after the record was fetched and looking at the disk if the file exists.

[chanmix51]

I think maybe you should use a ConvertedResultIterator for this, you do not really need OO entities for this.

[mvrhov]

This means that I have to mix arrays and entities which is not nice. Having the ability to clear those entities would be nice. Also using pomm in running long running processes e.g queue processors would benefit from this.

[chanmix51]

This is the job of another model manager, it will pop in in 3.0