refresh intervals

[ilf]

Currently, there is only one urlfile (per cachefile?). That means that all feeds in there have to be refreshed at the same time - and in the same interval.

I would love to be able to configure different intervals for feeds, some hourly, some daily, some weekly.

Could this be done via multiple url-files with the same cache-file? And/or some form of daemon process?

Ideally, I would love to go through a list of feeds and refresh them in random intervals. This would stop leaking the entire urlfile to an attacker on the line. (Thanks to Snowden, we can be certain the NSA is using that data for building user profiles.) For a GnuPG keyring, this is done via tools like parcimonie and rewrites like parcimonie.sh and GPG maintenance.

[Digenis]

I think the word "attacker" doesn't stand here. Also, if someone's list of feeds can raise enough suspicion to cause problems, refreshing in random intervals wouldn't do. Solving this is too complex and out of the newsbeuter scope.

You can open another issue on customising the reload-time per feed. Surprisingly, it has no duplicate.

[ilf]

I think, in 2015, the word "attacker" stands everywhere on the internet. Remember, the Five Eyes even target Angry Birds Apps.

And yes, refreshing at random intervals over Tor does work a long way against that. This is exactly what the references parcimonie does:

parcimonie incrementaly refreshes a GnuPG keyring in a way that:

• makes it hard to correlate her keyring content to an individual;
• makes it hard to locate an individual based on an identifying subset of her keyring content.

parcimonie is a daemon that fetches one key at a time using the Tor network, waits a bit, changes the Tor circuit being used, and starts over.

I don't have a concrete proposal or patch, since I don't know the newsbeuter internals well enough, But I think the idea of "different intervals for feeds, some hourly, some daily, some weekly" could be a start, no?

[simonnagl]

In my opinion implementing a new dependency to Tor is out of the scope of newsbeuter. This is a "simple" great feed reader and no security application. Refreshing at random intervals makes only sense for most people with a deamon process. If you just use newsbeuter to look if there are new feeds you have to refresh all. Then random refreshing whould only make sense before starting the graphical frontend. The third enhancement would make sense for me: Implementing a way thant feeds could refreshed at different intervals. I could work on that issue if one of the core developers could give me the okay that it could be implemented. Two general suggestions:

Use special configuration files for different refreshing interfals

1. Add a new command to the configuration file which could add another configuration file: add-config-file (parameters: <path> [<reload-time>]). If reload time is empty global reload time whould be used.
2. Add a special "tag" two the urlfile: <url> [<tag> ...] [reload-time:<minutes>] [<tag> ...]

In my opinion the second suggetion fitts more although an option for multiple urlfiles whould be anoter cool enhancement.

[ilf]

Yes, newsbeuter should not depend on Tor, that was not my proposal. The "proxy" option and/or torsocks are enough for that part.

I like the idea of a daemon process for fetching feeds regularly and in the background. There are all kinds of further things that could be done if fetching and display are different things handled by different components. See also https://github.com/akrennmair/newsbeuter/issues/175

I also like your suggestion to refresh at different intervals. But AFAICT that would only work from inside an already running newsbeuter instance. I would love a generic way that would work without having to run the "frontend" itself, since I refresh my feeds via cron.

[Minoru]

parcimonie uses random intervals to hide the list of keys you have from the server you're requesting keys from. parcimonie.sh also triggers Tor chain re-creation before fetching a key, so every request probably goes through separate chain.

In context of Newsbeuter, it's questionable if you need to hide anything from the server: I'd bet a large sum of money on median number of feeds people request per server being 1. Random intervals might help people who request numerous feeds from the same server (don't want BBC to know you're interested both in US and China, eh? :) That's the only valid point I see on that front.

If one is worried about MITM attack on the wire between router/Wi-Fi hotspot and ISP, and:

• aren't using Tor, and:
  • are in a crowded place (e.g. public Wi-Fi), random update intervals might help, although that's questionable: if they're the only person requesting RSS feeds, they're screwed; if they're using HTTP, then Newsbeuter is identifiable through User-Agent header and they're screwed; possibly many more scenarios in the same vain;
  • they're the only person on the wire, they're screwed regardless of update intervals;
• are using Tor, random updates might marginally decrease chances of their requests being tracked down, because there's no spike in requests, but running relay (with or without exit node) is higher up the list of things to enhance untrackability.

If there's a MITM of one and their router, and they're not using Tor, they're screwed. For MITM, using third-party server to fetch one's feeds is actually a good idea.

Overall, I think adding a random_wait setting a la wget should be sufficient. It would affect only auto-update; r, R and -x reload will force-update all feeds right away. Maybe adding -x continuous-reload that will be just like running Newsbeuter but without TUI (that would be blocked by #175).

Per-feed update intervals or multiple urls files are overkills.

[ilf]

Overall, I think adding a random_wait setting a la wget should be sufficient. It would affect only auto-update; r, R and -x reload will force-update all feeds right away. Maybe adding -x continuous-reload that will be just like running Newsbeuter but without TUI (that would be blocked by #175).

I like both of these ideas!

[Minoru]

It just occurred to me that there are still more ways to identify you: ETag and Last-Modified headers. Just sayin'.

[ilf]

You are right with all your points. But my desire is not to be "bullet proof" against the NSA, because there is no such thing. But every single step to make things harder for them will work against other, not-so-skilled attackers on the way. Every bit counts.

[klartext]

Reading the first entry here, it seems the title would better have been: "One Cache-db per Feed". I thought to add that as feature-wish here very often.

This maybe can be doe externally with calling newsbeuter with -c and -u options. Nevertheless seperate cache-files per FEED would make sense. Easier export - just give away the cache-db for a certain feed, if you want to share the feed-contents with others...

[Minoru]

Reading the first entry here, it seems the title would better have been: "One Cache-db per Feed".

Please read all the comments before replying. Per-feed caches were suggested as a possibility, but the topic is really different refresh intervals for different feeds.

Easier export - just give away the cache-db for a certain feed, if you want to share the feed-contents with others...

IMHO it's not worth the trouble, but if you believe it is, please open a separate issue and we'll discuss it there.