search

mattpodolak / pmaw

A multithread Pushshift.io API Wrapper for reddit.com comment and submission searches.
MIT License
212 stars 28 forks source link

Successful Rate 0% when batch loading comments by ids #23

Closed dukesun99 closed 3 years ago

dukesun99 commented 3 years ago

I found a strange bug, basically when calling search comments, large max_ids_per_request will result in 0% successful rate and forever retry. Maybe the result is too long?

To reproduce:

comments = api.search_submission_comment_ids(ids="haucpf")    
comment_list1 = [comment for comment in comments]    
comments = api.search_comments(ids=comment_list1, max_ids_per_request=1000)    
comment_list_full = [comment for comment in comments]

The above code works if we change to max_ids_per_request=500.

mattpodolak commented 3 years ago

Hi @dukesun99 it looks like Pushshift has started returning 520s for requests with 1000 ids, I'll include an update in the upcoming v2 release to set the maximum value to 500, thanks for reporting this issue