Large Table plans

[kjetilk]

It seems that if Attean::Plan::Table gets large, it can be a problem, since the rows attribute of the plan object will hold a lot of data. In my case, it seems like several gigabytes will go in there.

Perhaps it should only be retrieved in the impl method as an iterator?

[kasei]

Do you want rows to be an iterator instead of an arrayref? Complicating that might be the fact that it would need to be a Attean::API::RepeatableIterator, since calling as_string would otherwise consume the iterator.

[kasei]

Could you write your own version of Attean::Plan::Table that uses an iterator (that wouldn't need to be repeatable) and create that in your caching planner?

[kjetilk]

I could write my own, but it seems to me that it is a problem many are likely to encounter, and so perhaps this would be better solved generically in Attean?

Also, I think it would be a bad idea to have as_string include all the results... If it contains many gigs, it surely would be an unpleasant surprise.

[kasei]

That's a fair point. On the other hand, I've found it invaluable to see the contents of a Table when debugging query plans. Perhaps we should have two variants of a table in the core. The question then is: which variant should be used for the built-in cases when a table is needed (e.g. VALUES)?

[kjetilk]

I don't know, but I suppose that in a lot of cases, the rows would reasonably be held in RAM, and so the current Table implementation is appropriate, as it is most likely the fastest if it makes sense to keep it in RAM.

[kjetilk]

Seems this could be closed? Any reason why it is open?

[kasei]

I think it just got forgotten. Closing it now. Thanks.