pushshift
commented
5 years ago I'm getting closed to getting all of the data organized but I've run into an issue. For the Germany National Set, I can see the tags information using this:
https://api.mediacloud.org/api/v2/tags/list?rows=100&tag_sets_id=15765102&search=Germany
tag_sets_id 15765102 is for the Geographic Collection but when I do a search for the country Columbia, I don't see anything returned. It's possible this may be other another tag set and I just haven't found it yet but was curious if you knew which tag sets id it is under.
Also, for the retweets, I'm searching using:
https://api.mediacloud.org/api/v2/tags/list?rows=100&search=Retweet%20Right
I'm not finding an exact match for "Retweet Partisanship Left" and "Retweet Partisanship Right" with tags.
hroberts
We are currently running feed scraping code that is a few years old. Feed scraping for us just means discovering the set of rss feeds that cover all syndicated stories for a given site url. So feed scraping for 'http://nytimes.com' should ideally return a set of rss feeds that combined include all of the stories published at nytimes.com.
We are currently running old perl code (https://github.com/berkmancenter/mediacloud/blob/master/lib/MediaWords/Feed/Scrape.pm) to do this work currently. When we validated it several years ago, it was about as good as a human at discovering feeds. I'm sure it is worse now as the internet has moved on.
Last year, we worked on a newly implemented python module (https://github.com/mitmedialab/feed_seeker) to replace the crusty old code, and it seemed to mostly work, but we never validated it or hammered on it enough to get it into production.
The approach of both modules is similar. They use some pretty simple heuristics to discover the feed urls. The look for link tags, look at likely locations, look for any url with some relevant keywords like 'rss' in it, and also spider for one or two levels to look for rss feeds in sub pages (many sites have an 'rss' page that lists all of the rss feeds but no rss feed urls directly on the home page).
The ideal behavior of the feed scraper is to return the minimal set of feeds that we think will include all stories published by the source. The perl module has a notion of 'default feeds', which is just some set of feed locations that it tries and, if it finds a valid feed at that location, assumes that all stories are at that location. So for exmple if we find valid rss at 'http://nytimes.com/feed' we just assume that that feed has all the stories for the source. If we don't find one of those default feeds, we just return all rss feeds that we find. Having lots of feeds for a given source is fine if that is what it takes to get all of the stories, but we don't want to have to download a hundred feeds for a given source if we know that all the stories are included in a single feed.
It is sometimes the case that a given url will include rss feeds that link to other sources. This is bad for us, because it results in us adding stories to a source that don't belong to that source (think a random blog including the nyt rss feed on its site for whatever reason). We guard against this in the perl code by only returning rss feeds whose domains differ from the domain of the source url if there are no feeds found with the same domain.
We would like to test how well the new python module works by validating how the feeds returned by the python module compare to the existing feeds in our system (generated by our old perl module and sometimes manually tweaked by a human curator). In cases in which the feeds returned by the new python module and the existing feeds in our system differ, you should manually look at the differences and try to piece together whether each set of results is returning all, some, or none of the stories for the given source and also whether it is returning feeds that include stories not belonging to the given source. There is no easy way to figure out with full certainty whether a given set of feeds is actually returning all stories within a source, but just do your best to eyeball the feeds to get a sense for whether they seem to have all the needed stories (if there's a functional feed called 'all stories' that has dozens of stories per day, we can assume it has all of the stories). The idea here is just to do a quick validation of how well the system is working.
Elapsed time performance is not nearly performance as accuracy. If it takes us an hour to scrape for the feeds for each source, that's fine as long as the scraping is accurate.
We should do this validation for up to 50 sources for each of the following media collections:
You should be able to get a random set of feeds from each of the above collections using the media cloud api.