Pagination

[sammy-SC]

It would be awesome to have pagination. For large data sets it is necessary.

[asciimike]

@sammy-SC yep, we're investigating the best way to do this (we'd really like a count aggregation function in the client first), as FirebaseUtil does it but not in an ideal way.

I think this was planned originally for right after authentication, as we'll need to add another array and some properties for tracking which items exist where.

[kermankohli]

Any updates on this? Also what would define a "large collection" - at what point does performance degrade due to a lack of paging?

[puf]

We're still working on other features. Pull-requests are always welcome.

[jhnferraris]

Any news about this feature?

[asciimike]

No, though I'm planning a refactor of the underlying data structure at some point and might get around to adding new features.

[kutakmir]

I see this as a huge limitation Firebase has. Seriously, this is making it useless for virtually any app with a longer list. @mcdonamp @jhnferraris @kermankohli @sammy-SC @puf I am new to Firebase and have very little time for this, but how about having a live session and put it together? I am available in CET working hours.

EDIT: feel free to ping me on skype: Haunebu512

[jhnferraris]

Hello,

We've tried doing our own pagination for Firebase. We've tried @timominous approach. See here (http://stackoverflow.com/questions/37430881/swift-ios-firebase-paging). Maybe this can help kick off the solution of the pagination problem.

[kutakmir]

Do you got it working correctly? Could you perhaps share the whole paginating solution with the rest of us?

[jhnferraris]

@kutakmir

Yes. We just followed the sample usage in the attached SO link.

var count = numberOfItemsPerPage

var query ref.queryOrderedByKey()

if startKey != nil {
  query = query.queryStartingAtValue(startKey)
  count += 1
}

query.queryLimitedToFirst(UInt(count)).observeSingleEventOfType(.Value, withBlock: { snapshot in
  guard var children = snapshot.children.allObjects as? [FIRDataSnapshot] else {
    // Handle error
    return
  }

  if startKey != nil && !children.isEmpty {
    children.removeFirst()
  }

  // Do something with children
})

The solution might depend on how the scrollable list would work but I think the same logic applies. Here's the pseudocode that we used:

1. If startKey is null, then it is page 1.
2. Somewhere in your model, save the key of the last of object of page 1 and set that to "startKey"
3. Use the value of startKey to get page 2, remove the first child since it will now be a duplicate with the last value of page 1.

[kutakmir]

How about the newly created, edited and removed items in the paginated list?

[asciimike]

I strongly recommend folks look at the FirebaseUtil Paginate API, as it has properly implemented all the child_* methods, and has a number of other useful features. Porting this over to iOS and Android is something that we'd love, and is something we think the community would benefit from--as Puf said, PR's are always welcome :)

[timominous]

@kutakmir Unfortunately, the logic provided in the SO link doesn't have listeners for events.

@mcdonamp I found this (https://github.com/firebase/firebase-util/issues/51) and would like to think the team is already working on it. :) I would have to re-study JS if I'd have some time to port it over.

[asciimike]

@timominous that is in reference to this issue (though it wasn't exposed at the time), so just stay tuned to this thread for the latest :)

[robertodias180]

any news on this?

[CalvinTp]

Any update on this? As this thread started 1 year ago, please note that this is a common issue faced by a lot of people and it would help if this is looked into with some urgency.

[kutakmir]

Anyone with good knowledge of JS? Maybe creating an Obj-C wrapper would be enough: http://nshipster.com/javascriptcore/

[davidseek]

I get how pagination works if I'm downloading only my own feed, but how would it work, if I'm downloading for example the "post feed" of all my "friends" like in Twitter for example or Facebook. How would I have to sort the Node then?

Currently i'm gathering the IDs of all the Friends and then a request on firebase.child(id).child(userposts), but i have no idea how to do paging in this case?

I'd be willing to give up on event listeners, even tho it's the whole point for me in using Firebase at the first place..., and store the changed data like "likes" or whatever locally...

[morganchen12]

Ideally you should paginate by limiting your query to some number of items. Then as your user scrolls to the end of the posts, you can change your query and download extra items as needed.

We don't want to add pagination to FirebaseUI in a form where it aggressively downloads everything onto the client ahead of time and then only displays small sections of the query.

[davidseek]

Well. That is what I have done with the single User feed. No problem here. I have added an "index" key to every post and query for that by adding +5. But I have one view with all the feeds from User's friends. Gathered by:

load user IDs and then load posts for each user ID... no idea how to do the pagination here. one idea was to put all the posts, of all users into one big node and sort for user IDs first and then for index. but i have only 17 testers currently and each has 100+ posts... that would be one node with 1.700 posts. ONLY FOR MY TESTERS... can't imagine that it would be best practise to gather like the whole content of the app first...

[CalvinTp]

Can someone help to create an wrapper in Objective-C for pagination? It is amazing that there are so many issues around this area and Firebase is not adding a wrapper for us.

Apart from that, does anyone knows whether Firebase would be adding an API to return the number of child instead of returning all the child data? This would help significantly as Firebase is heavily focused on the client side processing.

[coreywiley]

@morganchen12

morganchen12 commented on Dec 22, 2016: Then as your user scrolls to the end of the posts, you can change your query and download extra items as needed.

Is there currently any way to change your query? If so, can you show how?

[morganchen12]

Not yet, though it's a more distant feature I'd like to have for FirebaseUI Database.

[billrancho]

I just can't believe firebase missing such an essential part of collection concept. My app now is growing and chat function just loads like forever as I got to pull thousands of records at a time. Any news on this issue?

[morganchen12]

You should limit your queries. When the user scrolls up to the top, you can swap your query for a new one with a higher limit. This way you're only downloading as many messages as the user needs to see.

There's a lot of boilerplate required to do this, which will hopefully make its way into FirebaseUI as a feature in the near future.

[svidata]

@mcdonamp Is there any real progress on this?

@morganchen12 mentioned that this feature will be released in the near future. May you please elaborate on this? How near is that future given that the last post was since 8 months ago and the original post was since 2015? (I have tried to limit my queries as you have mentioned above, but this is not in any way efficient in my case.)

We would really appreciate a clear answer on this so we know how to set our expectations.

[asciimike]

@svidata I'd take a look at Firestore which offers pagination support. It's likely that this support will never be added to the existing Realtime Database (even client side in Firebase UI), though I believe it's already possibly in the FirebaseUI Firestore module.

[svidata]

@mcdonamp thank you for your quick reply. You are saying that this support is never likely to be added but issue #17 in android and issue #9 in iOS are still set as open as you can tell. How come we have not been informed about this in those threads? Now, this means that we are expected to move everything to Firestore? Given that Firestore is still in beta mode??

[asciimike]

Let me clarify. It's highly unlikely that anyone from the Firebase team will add support for this in the existing Realtime Database, given that it's 1) inefficient (done client side), and 2) we're putting more effort into Firestore (which does it more efficiently, server side).

FirebaseUI is an open source project, which means anyone can contribute this feature if they'd like (and ideally, do so on both platforms), hence issues being kept open. Happy to close these and let the community re-open (or submit a PR) if that would make folks happier.

[svidata]

@mcdonamp How about the fact that Firestore is still in beta mode and more expensive than realtime database (Document writes Document reads Document deletes)?

[asciimike]

There are several threads on related topics on Google Cloud Firestore Discuss, so I'd voice concerns there.

@morganchen12 (iOS), @samtstern (Android), and I are happy to review PRs for this feature, but again, it's unlikely we'll be able to contribute significant implementation.

[samtstern]

If anyone reading this wants to send a PR, as @mcdonamp said, that's great but please outline your design in an issue first. This is a really really tricky issue to get right (@morganchen12, @puf and me have all made some attempt) and I wouldn't want anyone to waste their time coding before knowing if they were heading in the right direction.

[tillchen]

Any news guys? It's 2020 now 😆

[cosmopolit]

Time is running faster.. now we are in 2022 and after reading this i am a bit disappointed because i must write my own soloution :)