pushshift / api

Pushshift API
1.29k stars 107 forks source link

Scraping subreddit comments from Pushshift API not working #134

Open MarieSrsr opened 1 year ago

MarieSrsr commented 1 year ago

I have seen several comments on this subject, does anyone have an answer why the API is not working anymore ?

I have been trying with this :

getPushshiftData2 = function (postType, ...) { if (postType == "comment") { getPushshiftURL(postType, ...) %>% jsonlite::fromJSON() %>% .$data %>% jsonlite::flatten(recursive = TRUE) %>% select(author, title, selftext, created_utc, id, num_comments, score, subreddit) %>% as_tibble() } else { getPushshiftURL(postType, ...) %>% jsonlite::fromJSON() %>% .$data %>% jsonlite::flatten(recursive = TRUE) %>% select(author, body, parent_id, score, created_utc, id, subreddit) %>% as_tibble() } }

and save the date in conversion date to epoch

And all I get is empty files with the same date. There seems also to be a problem with the conversion between date to epoch conversion.