getGroup - callApi: "An unknown error has occurred."

[pdelboca]

Hi everyone! First of all, thanks for all the work and the library. It's pretty cool.

I'm having an error while using the getGroup method. After retrieving X amount of posts it throws an error:

Error in callAPI(url = url, token = token) : 
  An unknown error has occurred.

I'm trying to retrieve all the posts of a secret group, I'm a member of that group and the creator gave me admin privileges. I'm using a temporary token (not an App) and I've assigned all of the permissions when I created it.

The problem doesn't have a concrete behavior. It happens after 300 posts, after 699 or after 40.000 posts. A few days ago it happened 3 times after reaching 40600 posts, today, it happened 5 times after 300. I could never get all the posts of the group it always fails before. It works perfectly if I retrieve only 50 posts. It happened with the CRAN version and the Github Version. I'm currently using 0.6.7.

The code is pretty straight forward:

token <- "character_token"
groupId <- "id_number"
posts <- getGroup(groupId, token = token, n = 10000)

I've tried to play with since, until, and n arguments to try to limit the amount of info but the error is still there. I've also tried to debug it a little. The method getGroup iterates 3 times, and the third time it goes into callAPI method throws the error.

The URL created by the method is:

https://graph.facebook.com/v2.0/<GROUP_ID>/feed?fields=from,message,created_time,type,link,comments.summary%28true%29,likes.summary%28true%29,shares&limit=100&since=2015%2F06%2F01&__paging_token=enc_AdBeUHOedlWPmezMv8J02dHUffZAsRe9UTy7Dt2E8JQsg63wDQjiga7LK6ZBiegPoLP1KZAUvkkp7xv3nq0019HhzLwBviKQiONNnGkfnxZAeNg6hwZDZD&icon_size=16&access_token=<ACCESS_TOKEN>&until=1443205178&access_token=<ACCESS_TOKEN>

And the error when pasting into the browser was:

{
   "error": {
      "message": "An unknown error has occurred.",
      "type": "OAuthException",
      "code": 1,
      "fbtrace_id": "AQHWn2WQg5N"
   }
}

I would appreciate further help. If someone need more info I can provide it.

Thanks!

[pdelboca]

I could finally get all the post from the Group although there is a strange behavior in the getGroup method and I will try to explain it. I don't understand why it is happening, but when using since and until parameters after retrieving some post (300 in my case) it creates an OAuthException.

These are the params created by the method: /feed?fields=from,message,created_time,type,link,comments.summary(true),likes.summary(true),shares&since=2015-06-01&until=2015-08-01&limit=100

Using those parameters in the Graph API Explorer from Facebook Developers Page will also end in an error after a few hundreds of posts. You can retrieve 100 posts and click in the paging.next link, but after a few pages, it will throw the error. Same with the while loop in the getGroup method, it will iterate correctly a few times before throwing the error.

I could retrieve almost all the posts in my group by using the n parameter without since and until only if I'm retrieving less posts that the total in the group, if not, the method will throw the same error when trying to retrieve the last posts. Example: If my group has 41160 posts, and I put n=42000, it will raise an error when trying to get the last 60 posts.

[pablobarbera]

Thanks for reporting this bug. I suspect it's the same error as in #62 since the API endpoint is the same. Some of these queries appear to trigger an API response larger in size than what the API can handle. I changed the default so that it now paginates in batches of 25 posts, and after some tests that should solve the issue at the expense of making the function slower. In any case, for any large data collection, I recommend doing it inside a loop with since and until to make sure you retrieve all the data. I'll close this for now, but if you're still having these issues after updating to the most recent version on GitHub, please feel free to comment again and I'll keep looking into it.

[haseebmahmud]

I am using the version 0.6.15. I am having errors like this,

> bdnews <- getPage("bdnews24", token=fbauth, n = 10000)
Error in callAPI(url = url, token = token, api = api) : 
  An unexpected error has occurred. Please retry your request later.
> bdnews <- getPage("bdnews24", token=fbauth, n = 10000)
25 posts 50 posts 75 posts 100 posts 125 posts 150 posts 175 posts 200 posts 225 posts 250 posts 275 posts 300 posts 325 posts 350 posts 375 posts 400 posts 425 posts 450 posts 475 posts 500 posts 525 posts 550 posts 575 posts 600 posts 625 posts 650 posts 675 posts 700 posts 725 posts 750 posts 775 posts 800 posts 825 posts 850 posts 875 posts 900 posts 925 posts 950 posts 975 posts 1000 posts Error in callAPI(url = url, token = token, api = api) : 
  An unexpected error has occurred. Please retry your request later.
> bdnews <- getPage("bdnews24", token=fbauth, n = 10000)
25 posts 50 posts 75 posts 100 posts 125 posts 150 posts 175 posts 200 posts 225 posts 250 posts 275 posts 300 posts 325 posts 350 posts 375 posts 400 posts 425 posts 450 posts 475 posts 500 posts 525 posts 550 posts 575 posts 600 posts 625 posts 650 posts 675 posts 700 posts 725 posts 750 posts 775 posts 800 posts 825 posts 850 posts 875 posts 900 posts 925 posts 950 posts 975 posts 1000 posts 1025 posts 1050 posts 1075 posts 1100 posts Error in callAPI(url = url, token = token, api = api) : 
  An unexpected error has occurred. Please retry your request later.
> fbauth <- fbOAuth(app_id="1779336898952771", app_secret="f20ee29b7f9aca919b06070f50bf30fd",extended_permissions = TRUE)
Copy and paste into Site URL on Facebook App Settings: http://localhost:1410/ 
When done, press any key to continue... 
Waiting for authentication in browser...
Press Esc/Ctrl + C to abort
Authentication complete.
Authentication successful.
> bdnews <- getPage("bdnews24", token=fbauth, n = 10000)
25 posts 50 posts 75 posts 100 posts 125 posts 150 posts 175 posts 200 posts 225 posts 250 posts 275 posts 300 posts 325 posts 350 posts 375 posts 400 posts 425 posts 450 posts 475 posts 500 posts 525 posts 550 posts 575 posts 600 posts 625 posts 650 posts 675 posts 700 posts 725 posts 750 posts 775 posts 800 posts 825 posts 850 posts 875 posts 900 posts 925 posts 950 posts 975 posts 1000 posts 1025 posts 1050 posts 1075 posts 1100 posts 1125 posts 1150 posts 1175 posts 1200 posts 1225 posts 1250 posts 1275 posts 1300 posts 1325 posts 1350 posts 1375 posts 1400 posts 1425 posts 1450 posts 1475 posts 1500 posts 1525 posts 1550 posts 1575 posts 1600 posts 1625 posts 1650 posts 1675 posts 1700 posts 1725 posts 1750 posts 1775 posts 1800 posts 1825 posts 1850 posts 1875 posts 1900 posts 1925 posts 1950 posts 1975 posts 2000 posts 2025 posts 2050 posts 2075 posts 2100 posts 2125 posts 2150 posts 2175 posts 2200 posts 2225 posts 2250 posts 2275 posts 2300 posts 2325 posts 2350 posts 2375 posts 2400 posts 2425 posts 2450 posts 2475 posts 2500 posts 2525 posts 2550 posts 2575 posts 2600 posts 2625 posts 2650 posts 2675 posts 2700 posts 2725 posts 2750 posts 2775 posts 2800 posts 2825 posts 2850 posts 2875 posts 2900 posts 2925 posts 2950 posts 2975 posts 3000 posts 3025 posts 3050 posts 3075 posts 3100 posts 3125 posts 3150 posts 3175 posts 3200 posts 3225 posts 3250 posts 3275 posts 3300 posts 3325 posts 3350 posts 3375 posts 3400 posts 3425 posts 3450 posts 3475 posts 3500 posts 3525 posts 3550 posts 3575 posts 3600 posts 3625 posts 3650 posts 3675 posts 3700 posts 3725 posts 3750 posts 3775 posts 3800 posts 3825 posts 3850 posts 3875 posts 3900 posts 3925 posts 3950 posts 3975 posts 4000 posts 4025 posts 4050 posts 4075 posts 4100 posts 4125 posts 4150 posts 4175 posts 4200 posts 4225 posts 4250 posts 4275 posts 4300 posts 4325 posts 4350 posts 4375 posts 4400 posts 4425 posts 4450 posts 4475 posts 4500 posts 4525 posts 4550 posts 4575 posts 4600 posts 4625 posts 4650 posts 4675 posts 4700 posts 4725 posts 4750 posts 4775 posts 4800 posts 4825 posts 4850 posts 4875 posts 4900 posts 4925 posts 4950 posts 4975 posts 5000 posts 5025 posts 5050 posts 5075 posts 5100 posts 5125 posts 5150 posts 5175 posts 5200 posts 5225 posts 5250 posts 5275 posts 5300 posts 5325 posts 5350 posts 5375 posts 5400 posts 5425 posts 5449 posts 

That means, I have to keep trying again and again until the posts are scraped completely. Two things I would like to report here,

1. Since the error shows up at irregular points, we can not decide after which point I should make a break. So reducing the amount of data to be scraped may not work every time.
2. I have tried a python script. That works. So I guess, it is not FBs limitation.