search

shaikhsajid1111 / facebook_page_scraper

Scrapes facebook's pages front end with no limitations & provides a feature to turn data into structured JSON or CSV
https://pypi.org/project/facebook-page-scraper/
MIT License
242 stars 67 forks source link

Error when scraping over 1000 posts #70

Open KBrianNgeno opened 1 year ago

KBrianNgeno commented 1 year ago

I'm trying to scrape a lot of data for a time series visualization. It crashes after some time and I get this error. Traceback (most recent call last): File "/home/brian/Attachment/KeNVIP Facebook Scrapper/fbScrapper.py", line 27, in <module> json_data = meta_ai.scrap_to_json() File "/home/brian/Attachment/KeNVIP Facebook Scrapper/env/fb_scrapper/lib/python3.10/site-packages/facebook_page_scraper/scraper.py", line 101, in scrap_to_json self.__find_elements(name) File "/home/brian/Attachment/KeNVIP Facebook Scrapper/env/fb_scrapper/lib/python3.10/site-packages/facebook_page_scraper/scraper.py", line 185, in __find_elements all_posts = Finder._Finder__find_all_posts( File "/home/brian/Attachment/KeNVIP Facebook Scrapper/env/fb_scrapper/lib/python3.10/site-packages/facebook_page_scraper/element_finder.py", line 357, in __find_all_posts Utilities.__close_driver(driver) AttributeError: type object 'Utilities' has no attribute '_Finder__close_driver'. Did you mean: '_Utilities__close_driver'?

shaikhsajid1111 commented 1 year ago

Can you share which version you used, and what code?

KBrianNgeno commented 1 year ago

I'm not sure about the version but I downloaded it this past Friday. Here's my code: `from facebook_page_scraper import Facebook_scraper

page_name = "MinstryofHealthKE" posts_count = 5_000 browser = "chrome" timeout = 6_000 #seconds headless = False scraped_data = Facebook_scraper(page_name, posts_count, browser, timeout=timeout, headless=headless)

call the scrap_to_json() method

json_data = scraped_data.scrap_to_json()

filename = "data_file" directory = "./" scraped_data.scrap_to_csv(filename, directory)`

shaikhsajid1111 commented 1 year ago

I will look into this but if you want that amount of posts that's difficult to do it in unauthenticated way, crawler will get blocked very easily

KBrianNgeno commented 1 year ago

Okay. Thank you.

saiganeshyadav commented 5 months ago

Hi @shaikhsajid1111 , @KBrianNgeno i have developed a code which can scrape more than 15 thousand posts easily, please reach out to me if you need any help on scraping bulk data from facebook.

KBrianNgeno commented 5 months ago

Hi @shaikhsajid1111 , @KBrianNgeno i have developed a code which can scrape more than 15 thousand posts easily, please reach out to me if you need any help on scraping bulk data from facebook.

Hi @saiganeshyadav . Does it work without authentication?

saiganeshyadav commented 5 months ago

Hi @shaikhsajid1111 , @KBrianNgeno i have developed a code which can scrape more than 15 thousand posts easily, please reach out to me if you need any help on scraping bulk data from facebook.

Hi @saiganeshyadav . Does it work without authentication?

@KBrianNgeno yes, it works.