kevinzg / facebook-scraper

Scrape Facebook public pages without an API key
MIT License
2.39k stars 628 forks source link

Limited number of comments #297

Open ghost opened 3 years ago

ghost commented 3 years ago

Hello! I wanted to ask everyone why with this simple code:

comm=[]
for post in get_posts('nintendo', pages=2,options={'comments':True}):
     comm.append(post)

does not return the entire set of comments but only a small part? the number is significantly less than 5000 as explained in the documentation (the number returned is 159) ...What am I doing wrong?

neon-ninja commented 3 years ago

Hi, 5000 is an upper limit. If the post only has 159 comments, only 159 comments will be extracted. Also, you might want to pass cookies, as some comment extraction related functions (like extracting replies) require cookies or credentials

neon-ninja commented 3 years ago
posts = list(get_posts('nintendo', pages=2, options={'comments':True, 'progress': True}, cookies="cookies.txt"))
for post in posts:
    print(post.get("post_id"), post.get("comments"), len(post.get("comments_full") or []))

outputs

4168993589851729 438 194
4151037621647326 98 43
4148574141893674 101 70
4144965658921189 533 248
4141878672563221 12 10
4128969113854177 47 0
mardrago commented 3 years ago

Also when I add the cookeis file and run your own code:

posts = list(get_posts('nintendo', pages=2, options={'comments':True, 'progress': True}, cookies="cookies.txt"))
for post in posts:
    print(post.get("post_id"), post.get("comments"), len(post.get("comments_full") or []))

it gives me this result:

4168993589851729 0 30 
4151037621647326 0 30
4148574141893674 0 30
4144965658921189 0 30
4141878672563221 0 10
4128969113854177 0 0
neon-ninja commented 3 years ago

What language do you have set on your account?

mardrago commented 3 years ago

What language do you have set on your account?

it

neon-ninja commented 3 years ago

Try switch to English. You'll need to re-export cookies after switching language

mardrago commented 3 years ago

Okay, thank you! Now the result is:

4168993589851729 0 229
4151037621647326 0 44
4148574141893674 0 70
4144965658921189 0 251
4141878672563221 0 10
4128969113854177 0 0
neon-ninja commented 3 years ago

What version are you using?

mardrago commented 3 years ago

where can i find the version?

neon-ninja commented 3 years ago

pip freeze | grep facebook