Pagination in Facebook GraphAPI data

[yunhwankim2]

Hi, all. I want to discuss pagination in Facebook GraphAPI data. According to GraphAPI documentation, there are two kinds of pagination; time-based pagination and offset-based pagination. My aim is to retrieve data from Facebook page of corporation and analyze it, so I studied chapter 2 of MTSW and it was great. But I failed to download next page of data.

I used "get_connection()" method as follows,

ACCESS_TOKEN = " ... (your access toke hear) ..." g = facebook.GraphAPI(ACCESS_TOKEN) ms_id = " ... (ID of Facebook page) ..." page = g.get_connections(ms_id, "statuses")

The code above got 'status' data that the administrator of the page wrote. But it got only around 25 statuses, and I used another command to retrieve next page, which is as follows,

page1 = g.get_connections(ms_id, "statuses", until=" ...(until number)... "

The (until number) can be found in "page['paging']['next']" which looks like "...&until=XXXXXXXX...". This is called time-based pagination.

However, when I tried to retrieve link data by below code,

page = g.get_connections(ms_id, "links")

it returns different paging result. There is no 'until' term, and there is 'after' term there. So I tried in a similar way,

page1 = g.get_connections(ms_id, "links", after="YYYYYYYYYY")

but it returned blank dictionary.... I think it is because of the difference between time-based pagination and offset-based pagination which seems to be used in links data. Does anybody know how to retrieve next-page data by offset-pagination?

Thank you in advance.

Yun

[ptwobrussell]

My understanding from the docs at https://developers.facebook.com/docs/reference/api/pagination/#offsetpaging is that offset-pagination is only preferred for when you want to slice into a result set:

---

Offset pagination can be applied to any Graph API endpoint in combination with any other type of pagination. It can be used on its own when you do not care about chronology and just want a specific number of objects returned.

Supports the following parameters:

offset : This offsets the start of each page by the number specified. limit : This is the number of individual objects that are returned in each page. A limit of 0 will return no results.

Note that if new objects are added to the list of items being paged, the contents of each offset-based page will change.

---

It reads to me as though you could slice into your time-based pages by providing additional parameters of limit/offset or you could use limit/offset to navigate through some "arbitrary" results.

Are you just trying to understand how to employ it, or is there something you're trying to do that you think requires it? Perhaps if I understood your use case better, I could be of more help. (But at the moment, the best I can do is try to interpret what I think the docs say without any further info.)

[yunhwankim2]

Thank you for the comment. When I get the 'paging' information from Graph API, there seems to be two types.

The first type is;

u'paging': {u'next': u'https://graph.facebook.com/20528438720/statuses?access_token=<>&limit=25&until=1339520175&paging_token=10150899318698721', u'previous': u'https://graph.facebook.com/20528438720/statuses?access_token=<>&limit=25&since=1377882272&paging_token=10151655487463721&__previous=1'}}

The second type is;

u'paging': {u'cursors': {u'after': u'MTAwMDAzMjYxMzk3ODE3', u'before': u'MTAwMDAxNTkwNzIyNjgz'}, u'next': u'https://graph.facebook.com/10150899318698721/likes?access_token=<>&limit=25&after=MTAwMDAzMjYxMzk3ODE3'}},

In the first case, it was not hard to download the next page. Simply typing "g.get_connections(20528438720, 'statuses', until='1339520175')" downloaded the data of the next page.

But in the second case, there is no 'until' term. I could see 'after' term and I tried "g.get_conncections(20528438720, 'links', after='MTAwMDAzMjYxMzk3ODE3'" but it returned the same data. I suspected that it was the only data that the page had, but I could see more data when I visit the page by web browser. I googled and read Facebook documentation, and I got to know about pagination things. What I want is just to download the contents, so there's no need for other paginations than time-based one if I can do what I want by time-based pagination.

Thank you again for the comment. I'm still trying to figure it out, and when I get the answer I'll share it here.

Yun

[ptwobrussell]

Just wanted to ping and see if you'd made any progress on this front?

[yunhwankim2]

Thank you for your interests. After about a week of googling and trials-and-errors, I got to know about below.

1) Among the three pagination options, cursor-based, offset-based, and time-based (refer to https://developers.facebook.com/docs/reference/api/pagination/), offset-based pagination seems to be the least used. I have read some blog posts which even said offset-based pagination is almost deprecated.

2) We cannot determine which pagination to use. It seems to depend on which content type we retrieve. Let me give my example. For my research project, I tried to download the contents of Microsoft Facebook page. So I used the command below.

g = facebook.GraphAPI(ACCESS_TOKEN) ms_id = " " # id of Microsoft Facebook page ms = g.get_connections(ms_id, 'posts', limit=25)

(If I use 'feed' instead of 'posts', it returns all contents including posted by other users than Microsoft page administrator.)

Then, I check whether it used time-based or cursor-based pagination by the command below.

ms['paging']

it returned something like below.

{u'next': u'https://graph.facebook.com/20528438720/posts?limit=25&access_token=""&until=1387828803', u'previous': u'https://graph.facebook.com/20528438720/posts?limit=25&access_token=""&since=1389128521&__previous=1'}

As you see, there is 'until' in the 'next' and 'since' in the 'previous' term. However, when I requested 'link' instead of 'posts' by command below, they used cursor-based pagination.

ms = g.get_connections(ms_id, 'links', limit=25)

{u'cursors': {u'after': u'MTAxNTE5MTYyNzM1NTM3MjE=', u'before': u'MTAxNTE5MTYyNzM1NTM3MjE='}, u'next': u'https://graph.facebook.com/20528438720/links?limit=25&access_token=""&after=MTAxNTE5MTYyNzM1NTM3MjE%3D'}

As you see, there are 'after' and 'before' in cursor-based pagination. I have tried to find out whether there is any way for me to choose the pagination option, I still did not succeed.

3) Once we know which pagination option is used, it is very easy to traverse through whole data pages by commands below. Only thing we need is just to give another argument. In case of time-based pagination, we can use

ms = g.get_connections(ms_id, 'posts', limit=25, until='1387828803')

, or in case of cursor-based pagination, we can use

ms = g.get_connections(ms_id, 'links', limit=25, after='MTAxNTE5MTYyNzM1NTM3MjE%3D')

These are all I have found so far. I think it is enough to download all the data we need from Facebook page and I succeeded to get the data I need from Microsoft Facebook page.

Hope these will be helpful.

Yun

[ptwobrussell]

Yun - thanks so much for this followup. If it's alright with you, I'll go ahead and close this issue but comment back on it if I can find any additional information that helps here. Please feel free to do the same. I really appreciate that you took the time the capture this here. I need to take a few minutes and work through it myself...

[yunhwankim2]

All right. Thank you for your interests and comments and I'll keep an eye on this issue.

[ojasdeshpande]

Hey @p4718, @ptwobrussell did you guys find any more information about how to retrieve next page using offset based pagination?

[yunhwankim2]

Hi @ojasdeshpande , thank you for your interest. After studying above, I could retrieve data I want from Facebook so I don't need to use offset-based pagination. Do you find any reason to use offset-based pagination other than time-based and cursor-based one?

[ojasdeshpande]

@p4718 I was just trying to get a list of all my fb friends. On using the "me/friends" query, I was getting a limited number of friends and the "next" link suggested that offset-based pagination was being used. Anyways, I later found out that one can't get all the list of all facebook friends (http://stackoverflow.com/questions/24704709/list-of-all-facebook-friends-python) and what I was doing instead was giving me the list of friends using the app. So for now, I am not working on that and have scrapped the idea. PS - I think that as there were very few friends who were using the app, that's the reason using the next link didn't give me any more results. It's not because offset-based pagination wasn't working, it was because all the results were covered in the first page itself.

[yunhwankim2]

@ojasdeshpande Oh, I see. My aim was to retrieve contents such as posts or comments, so I didn't try to do something about friends list. Since I saw that offset-based pagination was almost deprecated, I didn't use that kind of pagination. And other pagination options worked well. Thank you for informing something new.