search

scripting / Scripting-News

I'm starting to use GitHub for work on my blog. Why not? It's got good communication and collaboration tools. Why not hook it up to a blog?
120 stars 10 forks source link

How do feed readers store their data? #109

Open scripting opened 5 years ago

scripting commented 5 years ago

I've been working on a MySQL version of a RSS feed reader server app. Unlike River5, which stores all data in JSON files, this reader stores all data in a database, and accesses the data via queries.

In the last year, starting with Bingeworthy and then Feedbase, I've learned enough SQL, and have expert help when I need it. Not sure when or if I will release it. I wanted to understand the issues.

I find SQL is an awkward tool for accessing this kind of data. Which got me to wondering how other feed reader apps store their data. I'm supposed to be such an expert on this stuff, and I admit I don't know what they're doing. I wonder if some of them are using object databases.

jefftriplett commented 5 years ago

NewsBlur is pretty popular (open source Django app) and uses both Postgres and MongoDB. I linked the project in case the code is helpful. Here is main Feed model: https://github.com/samuelclay/NewsBlur/blob/master/apps/rss_feeds/models.py#L54

brentsimmons commented 5 years ago

NetNewsWire uses SQL for articles and read/unread/starred status. It stores feed/folder structure in an OPML file on disk, and it stores feed metadata in an object database.

brentsimmons commented 5 years ago

Here’s NetNewsWire’s SQL schema: https://github.com/brentsimmons/NetNewsWire/blob/master/Frameworks/ArticlesDatabase/CreateStatements.sql

scripting commented 5 years ago

Here's the schema for the feed reader I'm working on..

https://github.com/scripting/Scripting-News/blob/master/misc/feedDatabaseTables.md

Very interesting to compare structures.

mblaney commented 5 years ago

I run a reader at https://unicyclic.com which uses MySQL. schema: https://gitlab.com/dobrado/dobrado/blob/master/install/Reader.php#L343

ezraball commented 5 years ago

I'm not sold on NoSQL in general, but I've played with CouchDB a bit and its multi-source replication is very slick. If I were building a feed reader where a requirement was to sync multiple devices (for read/unread state) I'd consider it.