IFTTT / polo

Polo travels through your database and creates sample snapshots so you can work with real world data in development.
http://ifttt.github.io
MIT License
775 stars 48 forks source link

Stream queries when traversing database to reduce memory footprint #67

Open aamirsahmad opened 9 months ago

aamirsahmad commented 9 months ago

AFAIK Polo relies on ActiveSupport::Notifications.subscribed to traverse the ActiveRecord tree. This works fine when associated data is small however in production scenario, we have experienced huge memory consumption as collect_sql stores everything in memory. I was wondering if others have similar issues and how they solved this? Also, are there plans to perhaps implement a streaming style methodology (e.g. probably using yield) in future?