Is on-screen location dependent on index in the IGListDiffable array?

[sidmani]

New issue checklist

• [x] I have reviewed the README and documentation
• [X] I have searched existing issues and this is not a duplicate

I'm trying to use IGListKit with a large data set of memory-expensive items, and I want to load objects into the diffable array as they're needed to conserve memory. Let's say the immutable IGListDiffable array looks like this to start with:
[Object1, Object2, Object3, Object4]
where Object2 and Object3 are visible on screen. When the user scrolls, I supply a new immutable array:
[Object2, Object3, Object4, Object5]
because Object3 and Object4 are now visible, and Object1 is irrelevant. By doing this, IGListKit has no idea how many Objects I actually have, and I guarantee that at most N objects are ever in memory. Will shifting indices like this cause IGListKit to make cells jump around in the collection? Is there a built-in way to handle this? (I looked through the docs and saw IGListMoveIndex, which sounds relevant, but the docs don't say what it's used for). Thank you!

[rnystrom]

@sidmani oof, this is definitely a limitation. If objects have big memory footprints, and there are lots of them, this will become a problem.

Objects in the array directly map to their position within the UICollectionView. So if you remove Object1, like in your example, views will shift.

A loooong time ago, when I was working on AsyncDisplayKit, we had the idea of an "allocation range", where we could purge items from memory that were way out of the range of visible stuff. However, this is super duper complicated and we never came up with a great solution.

One idea for you, is maybe you can replace Object1 with some sort of placeholder object? I'm not sure how great that'd be, but its an idea.

Can you give any more details on what is so memory-heavy about your data? There might be another solution, like using a purgeable store to hold onto the heavy stuff, and looking up that data via some sort of relationship or edge, like a uuid.

[sidmani]

The objects represent messages, with a 2500 character limit. This means that each message could potentially become as large as 5-7kb due to the text (depending on encoding). There is no limit to the total number of messages, but having 500-1000 messages in a conversation is completely reasonable, so the worst case scenario involves more than a megabyte or two (!) of text data, per conversation.
Second, as the length of the array increases, using filter and map to search for and update data in the array becomes more and more expensive.
Perhaps I'm overreacting to an edge case that would likely never happen, but keeping the excess data out of memory ensures that the app doesn't progressively slow down as the # of messages increases, and keeps scrolling completely smooth (assuming I can load messages as fast as the user scrolls). It seems like making the message text immutable (so diff doesn't need to consider it) and using wrapper objects containing metadata and a UUID is the way to go.

[rnystrom]

Awesome details! Here's some advice:

• Definitely use an identifier and make the contents immutable. Comparisons are then reliable and cheap, no need to check equality of the contents.
• Any filtering and updating could be moved to a bg queue if it's measurably expensive
• To me, a couple mb of text data is nothing. A single decompressed image will be way more than that.

I would def try to avoid over optimizing without any measurable perf issues. Requirements might change, or what you might think is an issue actually doesn't matter.

[sidmani]

Thanks! I'll do that, then.