search

drewmccormack / ensembles

A synchronization framework for Core Data.
MIT License
1.63k stars 131 forks source link

Memory warnings and crashes due to memory pressure #114

Closed jefferybradley closed 10 years ago

jefferybradley commented 10 years ago

Thank you for providing an iCloud sync solution that "just works"!

I would like to ship my next update with Ensembles enabled, but I am running up against a memory pressure crash while [CDEEventIntegrator integrate:] is executing, after importing/processing a large baseline (XCode lists the baseline at 27 MB).

My application is a research note taking app, so these notes can be fairly lengthy. And I have users with collections of over 1,000 notes, so batching during integration or processing smaller baselines is much needed. When a new device joins the Ensemble, it has to process the entire baseline at once and once Ensembles reaches the content of the notes, memory spikes, and the crash quickly follows. Internally, the app processes notes in batches so I was wondering if there is a method, at the moment, to ask/tell Ensembles to process/integrate Entities in batches?

If not, any ideas on how I can approach the memory issues until Ensembles features more memory optimizations? Will this problem be resolved when issues #36 and #104 are addressed?

And again, thanks so much for your work. Ensembles is a godsend!

drewmccormack commented 10 years ago

Thanks for the feedback. Yes, this is something we definitely need to address. Probably the last major piece of the puzzle. Many (most?) developers will never run up against the issue, but anyone with a large library, particularly on iOS, will have this issue.

Note that large NSData attributes are now properly handled as external files, so they should not be a problem. If you have large NSData attributes, and aren't using the very latest master branch version, you should upgrade. I only finished that a week or two ago.

If on the other hand you just have a lot of data in the DB, you may need to wait. I can't think of any other solution, short of splitting your data in to multiple stores, or perhaps storing parts of your model as NSData using transformable properties, but neither are good solutions.

Addressing this problem is a priority now. I will be looking at it in the coming weeks/months.

If you want to help the process, you could post instruments profiles for the runs that crash. It might also help to email me your model if you are prepared to do that. You can send it privately to drewmccormack at mac com

jefferybradley commented 10 years ago

Thanks for the input. Incidentally, I am working on storing the text data as NSData to take advantage of how the latest master branch handles large NSData attributes and because I cannot reliably predict the length of users' individual notes. Is Ensembles measuring the size of the NSData before handling as an external file (to determine which are large) or is every NSData attribute handled this way?

drewmccormack commented 10 years ago

Yes, it only stores large NSData objects externally. Currently the cutoff is 10KB I think. We could perhaps make that configurable if that is an issue.

The relevant code is in storeAttributeChangeForDescription:

drewmccormack commented 10 years ago

To followup, I guess we could also add large strings to our external file approach. That would save you and others having to change the model to turn strings into NSData attributes, and it would not be a big change to the framework.

Is the size of your DB largely down to large NSStrings? What are typical sizes?

drewmccormack commented 10 years ago

Actually, maybe it wouldn't even help. What would probably happen is that the framework would use the NSData to create a string, and it would do that in memory. So it wouldn't take advantage of memory mapping or anything else. So I doubt storing strings externally would be much use.

At this point, you best bet is probably to use NSData attributes for the strings. They should only need conversion to NSString when you use them, I guess.

jefferybradley commented 10 years ago

Thanks.

Both the string data and the number of records is an issue. Combined, the DB can grow relatively large.

The text is stored as NSAttributedStrings using a transformable attribute. But I am testing using data that measures about 3K - 4K. Though I know some users are storing much longer notes than that. But I have to test using datasets of 500 - 2000 records each with 3K-4K in data since I have users with well over 1,000 notes of varying length. Dedicated users are more likely to generate hundreds of note records.

I will send the model to you so you can take a look.

drewmccormack commented 10 years ago

We are working on this for a future update now.