search

hyperoslo / mingle

Social media integration for Ruby on Rails
http://rubygems.org/gems/mingle
Other
20 stars 2 forks source link

Clean up instagram removed items #28

Closed sergiorivas closed 3 years ago

sergiorivas commented 9 years ago

Clean up instagram removed items consists in verifying deleted photos doing a http (head) request for stored photos, You have to check this after fetching photos. I try to do the process more efficient ignoring recently fetched photos.

jgorset commented 9 years ago

Thanks, @sergiorivas! That's a nice touch with ignoring the most recently fetched photos.

It's not uncommon to maintain a database of thousands of images with Mingle, though. That's thousands of queries every time you fetch a new batch of images (usually every few minutes). Do you have any ideas as to how we could avoid that?

sergiorivas commented 9 years ago

I think we can externalize the method verifying_integrity_for_deleted_photos in order to be invoked from a background task like a cronjob. Then, you can delete removed photos in background and improve the performance. This idea could be applied to another context.

jgorset commented 9 years ago

That seems a reasonable approach, @sergiorivas. As it is this feature isn't really feasible due to performance constraints, so I'll leave this pull request unmerged pending a change like you describe.

jgorset commented 9 years ago

Just to be clear, I'm not expecting you to make that change — it's up for grabs, and this is a nice starting point.

sergiorivas commented 9 years ago

I removed the optimization to be more general, but now the prune method can be called from a cronjob task.