search

minimaxir / facebook-page-post-scraper

Data scraper for Facebook Pages, and also code accompanying the blog post How to Scrape Data From Facebook Page Posts for Statistical Analysis
2.12k stars 663 forks source link

until date issue #61

Open iamrajeshchauhan opened 7 years ago

iamrajeshchauhan commented 7 years ago

Hello Minimaxir,

In python code line no. 187 showing some error while running with since and until date.

code : if 'paging' in statuses: next_url = statuses['paging']['next'] until = re.search('until=([0-9]?)(&|$)', next_url).group(1) paging = re.search( '__paging_token=(.?)(&|$)', next_url).group(1)

error :

Traceback (most recent call last): File "Fbpost.py", line 199, in scrapeFacebookPageFeedStatus(group_id, access_token, since_date, until_date) File "Fbpost.py", line 187, in scrapeFacebookPageFeedStatus until = re.search('until=([0-9]*?)(&|$)', next_url).group(1) AttributeError: 'NoneType' object has no attribute 'group'

can anybody help me what is the issue while running this new Facebook page python code

Thanks

gabru-md commented 7 years ago

@iamrajeshchauhan What is happening is that the re.search command in line 187, i.e.

until = re.search('until=([0-9]?)(&|$)', next_url).group(1)

is returning no results.

And since there are no results produced, the return type of the re.search() command will be NoneType and NoneType has no attribute called group in it.

Hope it helps! :)

iamrajeshchauhan commented 7 years ago

@gabru-md So while running code what should I do?

gabru-md commented 7 years ago

what page are you trying to scrape?

iamrajeshchauhan commented 7 years ago

it's not fixed, so in the code what changes are we require to solve the error. You can take any page example for scrapping.

minimaxir commented 7 years ago

Huh, I tested this workflow and didn't see an error. A possible cause is that if there isn't a next_url.

A workaround may be to only set until if no defined until exists.

iamrajeshchauhan commented 7 years ago

great ! thanks, @minimaxir Is that possible can we scrap the data(comments, reply, likes, etc) from Facebook where the post was live video?

LinuxBeginner commented 7 years ago

@minimaxir Hi, I am getting the same error. I gave since_date = "2017-01-01" # ( 1st Jan 2017 ) until_date = "2017-07-02" # ( 2nd July 2017) . The error comes just after 100 records (1st July 2017 back to 8th Feb 2017) are written on the .cv file. Post are available in the page for Jan 2017 also.

tiborko commented 7 years ago

Hi @minimaxir I am experiencing the same issue. I only get the first 100 records. Thereafter I get the same error. I know there are many more. I tried different dates, always get 100 records only. I am not sure how to implement your suggestion from June 20th, above. Any suggestions, please?

iamrajeshchauhan commented 7 years ago

Hi @tiborko ,

I think if you can use below script for scrapping that would be great.

https://github.com/minimaxir/facebook-page-post-scraper/blob/master/get_fb_posts_fb_page.py

Thanks !