kevinzg / facebook-scraper

Scrape Facebook public pages without an API key
MIT License
2.35k stars 626 forks source link

Problem with groups having top post #635

Open qducnguyen opened 2 years ago

qducnguyen commented 2 years ago

Hi, Thanks for your delication to this project, but I have problem with the group that has the top post, I just scraped max 21 post in this group. So, do you have the solution for this? Thanks in advance.

Here is the code I took from the issue #310 and with the group ID having the top post

    def get_current_search_page(self) -> Optional[str]:
        return self.search_page_url

    def set_search_page(self, page_url: str) -> None:
        print('Storing next search page {}..'.format(page_url))
        self.search_page_url = page_url

     search_page_persistor: SearchPagePersistor = SearchPagePersistor()  # could be inited with a specific search page URL 

    for post_idx, post in enumerate(get_posts(
           group='418033225381445',
           cookies=FOLDER_PATH + 'cookie.json',
           page_limit=None,  # try to get all pages and then decide where to stop
           start_url=search_page_persistor.get_current_search_page(),
           request_url_callback=search_page_persistor.set_search_page
    )):
       print(post)

And here the output I get, It abruptly stopped at the top post .

 Storing next search page https://m.facebook.com/groups/418033225381445/....
 {'post_id': '1295173794334046', 'text': 'Bán đất gần Bến xe ....
 {'post_id': '1373456766505748', 'text': 'Đất đẹp Liên Mạc
  ......
  21 posts
 Storing next search page https://m.facebook.com/groups/418033225381445?bac=MDoxMzczOTUzNzIzMTIyNzE5OjEzNz..
 {'post_id': '1295173794334046', 'text': 'Bán đất gần Bến xe ....
neon-ninja commented 2 years ago

Try load up https://m.facebook.com/groups/418033225381445 in your browser, and scroll down. It doesn't load more posts. This is a bug in Facebook itself, not a bug in facebook-scraper. Duplicate of https://github.com/kevinzg/facebook-scraper/issues/532