RSSTootalizer - Rate Limiting and Digest

[mj-saunders]

Feel free to post additions via comments; we can always improve what follows.

Potential options to pursue:

1. Display every n^th RSS post - not ideal
2. Toot a digest of n posts per toot
3. Use a hashtag or keyword list - but who decides? And I guess the bot would have to somehow publish and be clear what tags it is searching for and posting.

Details

Option 2 - Toot Digest

This appears to be our best option for the time being. For purposes here, n = 5

Method 1

• Wait for 5 posts to accumulate, then toot
  • Reset a 24hour timer
• Toot a digest once every 24hours, even if 5 has not been reached
• Allow more digests per day?
  • How many?
  • How to space these throughout the day? ¹
• Allow for rollover of posts beyond the daily limit, so they have another chance to be tooted the following day
  • How to stop this from creating massive delays eventually with high-volume bots? ¹

Method 2

• Check RSS every hour
  • Should this be less frequent? If so, do we risk too much delay?
• If there are more than one post, toot a digest (otherwise do normal toot)
  • up to a maximum of 5
• If there are more than 5, rollover to next hourly check
  • if next check yields again more than 5, how to prioritise?
• This potentially leads to 24 toots per day for a bot
  • Too much?
  • How best to rate-limit?

¹ I dislike this as we are then altering when content is being published

I am sure there are more issues. Please comment and question so we can improve, decide and solve this quickly 😄

[Openmedianetwork]

OK the is no perfect option, from a technical point of view i prefer each link to be a separate toot. BUT this is not good from a user point of view.

POTP 1) not ideal 2) maybe the best option 3) we are doing a bigger project to make this idea work http://omn.openworlds.info but its not going to be in a usable state for a year or more, so no.

M1 Think M2 better. Maybe have a configure setting for 3/5 toots for different toot flows

M2 Think maybe we should try this one? Check every hour then toot as a group up to 5 or normal single toot

we mediate this by just not adding any feeds that are to fast flowing

[mj-saunders]

OK. I agree - prefer M2.

I shall spend a little time going over the existing code, to see how best to implement this.

Will also allow for today, for any of us to come up with a better solution before I start hacking at code, and for the voice of a certain Vagabond to be added - I like consensus.

[mj-saunders]

Notes (mainly) to self - feel free to ignore this post:

• Check RSS every hour

Already done with cronjob - could tweak frequency using crontab -e

• If there are more than one post, toot a digest (otherwise do normal toot)
  • up to a maximum of 5
• If there are more than 5, rollover to next hourly check
  • if next check yields again more than 5, how to prioritise?

Perl code edits. I believe this should all be achievable without altering the database structure.

• This potentially leads to 24 toots per day for a bot
  • How best to rate-limit?

Even though cronjob checks feeds every hour, perhaps rate-limiting should be done via Perl. Would allow for timely checking, but not necessarily such frequent posting. RSSTootalizer originally intended to use the cronjob to determine posting frequency.

TODO: Analyse how the db is utilising timecodes and decide how to proceed.

[mj-saunders]

a functioning (prototype?) is now running utilising most of Method 2