Memory overflow issues when dealing with large datasets.

[wmelton]

I have been using and loving this library for a while now. I ran into a unique situation this week where I needed to iterate over a very large dataset and for each row perform unique logic based on varying key factors. Since idiorm creates id keyed data objects for each row of data matched with 'find_many' or 'find_result_set'(or maybe I don't understand how find_result_set works) and stores them in memory - what is your suggestion to best iterate over large datasets using idiorm without bringing 5k rows of data up in to memory to be iterated through all at one time, or would you say that's really outside of the realm of what idiorm was meant for?

I found a less than elegant work around to be able to deliver in a pinch, but this approach is going to have to be refactored because it results in ALOT of reads from the database.

while( ORM::for_table('someTable')->where('uniqueIdentifier', 'someKey')->find_one() ) 
{
 //Do some stuff with the row and ->save() until we run out of matching rows;
}

Any opportunity to create a method that returns just an array of ID's that are iterable with logic to 'update where id = ' upon save() instead of bringing the entire data object into memory? I'd be more than happy to prototype the pull request if you think there's room for this within the bounds of idiorm. Would be super helpful for mass updates.

If I've misunderstood any existing functionality, or a feature request somewhere else that already discussed this please forgive me.

Again - thanks for a VERY awesome and useful database library for PHP. Easily the simplest php db library I've used over the years.

[treffynnon]

Glad you like the library. Unfortunately Idiorm is not designed to work with this kind of row volume or in this way.

If you're updating this many rows at once I would drop down to PDO make the update directly in SQL.