Synchronize Agent Schedules

[level20peon]

I think it would be a good idea to put Agent Schedules on a cron like timetable rather than a timetable related to the Huginn launch time. So when I put an Agent on like "Every 12h", it will be scheduled for Midnight + Noon, rather than every 12 hours from the time when I launched Huginn. I know that this could be done by using SchedulerAgents, but that would mean one more Agent per scenario, which in my opinion would be overkill. What do you guys think?

[dsander]

I totally agree, it's been on my todo list for some time, will have a look at it on the weekend

[knu]

Most existing "every" schedules can be converted to a cron line, because those intervals are divisors of 86400 seconds (1 day). However, there are some other "odd" intervals that are hard to be converted: 5h, 2d and 7d. It is not clear how they should be aligned.

[knu]

Also, should it be fair if we fixed the 1d interval to a specific time of a day?

[level20peon]

Every 5h: * /5 * * Every 2d: 0 0 /2 * Every 7d: 0 0 /7 *

And maybe you haven't updated Huginn in a while, but there are schedules for specific times of a day :) Or maybe you meant that the 1d interval should have a predefined value, but in this case, it would be executed at midnight every day, supposed the fixed schedule I proposed would be implemented, no?

[cantino]

I agree. In addition, if Huginn gets restarted right now, all schedules that didn't run and are longer than the period get lost. For example, if your Huginn instance gets restarted every 5 days and you have an every 7 day schedule, it will never run.

[dsander]

@knu You are right that a cron schedule for those odd intervals is not perfect but at least you can depend on the date and time when its run. The "1d" interval is not exposed to the user, we have "noon" and "midnight".

We could change "7d" to "every monday" or add an "every X" option for every day of the week.

[virtadpt]

I think it's nice to not have entirely deterministic run schedules. For example, when trying to be a good netizen when hitting external API services (and trying not to get banned from them), not hammering them every ten minutes on the dot spreads out the load.

[level20peon]

Well that should be the responsibility of each user and not of the Huginn system. Also think about clocks on every system (probably) not being exactly synchronized, so no harm there :)

[level20peon]

Any news on this matter?

[level20peon]

Since this doesn't seem to get solved I would like to know if it's possible to run specific clients via console. I have a keepalive script which restarts huginn in case it crashes (and in some other circumstances) and would like this keepalive script to run specific agents for the first time on a specific time after starting huginn (like on 6am, 12am, 6pm, 12pm). For that I need a console command.

[cantino]

It'd be a bit hacky, but bundle exec rails runner "Agent.run_schedule('11am')" would run the 11am schedule, or bundle exec rails runner "Agent.async_check(123)" would run the check for Agent#123.

If you're on your production setup, you'd need to add RAILS_ENV=production before both of thise.

[level20peon]

Thanks, that looks promising. Could you elaborate on "would run the check for"... does it run the agent when I issue this command or does it perform some kind of check which might result in the agent not being ran?

[cantino]

It runs Agent 123's check action (that would normally run on a schedule). It doesn't trigger receive, which is what would happen if an event were coming in. You could also force event propagation if any events are waiting with bundle exec rails runner "Agent.receive!".

[level20peon]

If I issue this command for a Delay Agent, what would happen? Would it cause the Delay Agent to pass the cached events or would it just check for incoming events? I just need Delay Agents to be synchronized, every command achieving a result other than that, I cannot use in my scenario.

[cantino]

It looks like bundle exec rails runner "Agent.async_check(123)" on DelayAgent#123 would cause it to emit.

[cantino]

I also agree that a more cron-like system would be a good addition.

[level20peon]

ok here's my workaround (in case someone is interested) until a cron-like system will be implemented. I want my Delay Agents to be ran twice a day, on 06:00 and 18:00. So depending on when huginn gets started this script waits for the next occurence of either those times and then runs all Delay Agents. This script is launched by my keepalive script, which also launches huginn beforehand:

#!/bin/sh

current_epoch=$(date +%s)
current_hour=`date +"%H"`

if [ "$current_hour" -lt "6" ]; then
    target_epoch=$(date -d 'today 6:00' +%s)
elif [ "$current_hour" -ge "6" ] && [ "$current_hour" -lt "18" ] ; then
    target_epoch=$(date -d 'today 18:00' +%s)
else
    target_epoch=$(date -d 'tomorrow 6:00' +%s)
fi

sleep_seconds=$(( $target_epoch - $current_epoch ))
sleep $sleep_seconds

cd ~/huginn
RAILS_ENV=production
/usr/bin/mysql -N -h 127.0.0.1 -u user --password=password -e "USE huginn;SELECT id FROM agents WHERE type LIKE '%DelayAgent%';" | while read a; do
    bundle exec rails runner "Agent.async_check($a)" &
done

[level20peon]

Well the workaround doesn't work as intended. It does in fact run the Delay Agents at the time I want but the scheduler doesn't reset their timers when they get run. So let's say a Delay Agent currently has a 12h schedule which has its next iteration at 9am. Now I run all Delay Agents at 6am with the script above. The scheduler still runs said Delay Agent again at 9am (and then at 9pm, again at 9pm and so forth). So is there a way to reset the scheduler additionally to running the agent at a specific time?

[dsander]

Does it work when you change the DelayAgent to the "never" schedule and only have it run by your script?

[level20peon]

Probably, but I would have to run it by script all the time, which defeats the purpose of the built in scheduler. I'd rather "just" reset the built in scheduler somehow.

[level20peon]

ok, I created a workaround for the workaround. The launch/keepalive script now launches huginn, runs all Delay Agents on start (in order to propagate "old" events), then waits until the next occurrence of my target time (6am / 6pm) occurs and then simply terminates huginn and launches it again, and again runs all Delay Agents at that point. It's a workaround of a workaround, so I would rather really like to see a cron like schedule implemented.

[cantino]

@level20peon, I want to make sure that I understand the problem you're solving. Why didn't using the '9am' schedule work for you? Was this because you want it to run more than once per day, at a specific time? And what was the reason that ScheduleAgents wouldn't work for this? Was this because you'd need too many of them?

[level20peon]

The Agents which I want to "synchronize" have "every 12h" schedules, so yes, they are run multiple times a day (at 6am and at 6pm). I could use ScheduleAgents but I don't know if such setup would be more elegant than "just" synchronizing schedules when Huginn is launched !? Additionally, it's a hassle to keep track of hundreds of Agents being assigned in one ScheduleAgent or another.

[cantino]

Agreed, we'd definitely like schedules to be more cron-like on Huginn launch.

[dsander]

I tried this once, if I remember correctly the issue were very slow startup times of rufus when I changed our Xm/Xh schedules to a cronline. To make it work we might need to add some optimizations to rufus first.

[Jngai]

@cantino This ticket looks interesting. I will take a look.

[cantino]

The challenge here, I think, is to remember the schedules (or infer them from a fixed starting point in time?) so that if Huginn is restarted, 1/week schedules still run, etc. Does this make sense?