Create API for apps to schedule background tasks

[paulproteus]

Some apps like Tiny Tiny RSS expect to be able to run themselves in the background, periodically. Currently there is no way to do that (except hypothetically by holding a wakeLock for a long time, but that is very inefficient).

This ticket tracks the creation of some cron-like API by which apps can do that.

[pdurbin]

@paulproteus good on you for creating this issue. Here's where someone asked for this feature: http://logbot.g0v.tw/channel/sandstorm/2015-11-12

[ocdtrekkie]

My life will be so much more complete when this is done. Annoyingly, The Verge and Windows Central are examples of blogs whose RSS feeds present less than 24 hours worth of posts, meaning I miss a lot of stuff if I don't open my TTRSS grain multiple times throughout the day.

[ocdtrekkie]

@pdurbin It has also been on my Sandstorm Feature Wish List since May or before. :D https://alpha.sandstorm.io/shared/_CRj3LHU3vARmZ0-5DGQcnEzf1NrHcjULFaFaO9hslR

[jparyani]

An interesting stopgap solution just occurred to me. We could make a Sandstorm cron app that uses the backgrounding api to stay up all the time. It could then collect webkeys to other apps, and wake them up on the cron timer. Any thoughts?

The one annoying bit is that the backgrounding api currently doesn't persist through frontend restarts (see #595).

[ocdtrekkie]

That'd definitely work for TTRSS, since the mobile app's API calls force it to update.

Arguably, from the "make everything replaceable/make everything an app" standpoint, if it's possible/practical to develop cron as an app, isn't that potentially superior to a Sandstorm API to do a cron?

[funwhilelost]

Yeah, that wouldn't be a bad solution. People used to do this for Heroku's free tier that went sleep (just ping it occasionally).

Many apps will track their background work using Resque, Beanstalkd, RabbitMQ, etc. Is this a "service dependency" that Sandstorm supports yet? Would it be easier to implement a background job handled by implementing one of their protocols?

[fabacab]

I am also interested in this. I have been developing a WordPress plugin that is time-sensitive. WP-Cron isn't reliable on a low-traffic website, so my plugin programmatically tries to install a cron job to execute wp-cron.php every five minutes. That ensures my plugin doesn't miss an important event by more than 5 minutes.

Unfortunately, this doesn't work on Sandstorm. :( I'd love to contribute in whatever way I can to make this possible, but I am completely new to Sandstorm (really like it so far!) so I'm not sure how much I can feasibly do.

[mitar]

What about just having some flag to allow running grains permanently for now? I use this great Meteor package for my background jobs. It would be great if this could just work out of the box. But why the package is very cool is because it allows different ways to run a worker. So maybe there could be a special Sandstorm worker to run those jobs.

[kentonv]

@mitar Note that we want to be careful about making it too easy to consume lots of resources on Oasis. Our pricing is based on the assumption that people aren't using Oasis like they would a normal always-on server. Eventually we'll actually enforce limits on compute resources but we haven't had a chance to implement that yet...

[mitar]

I completely agree with not allowing CPU intensive applications. But here I am more thinking about personal "background workers/bots" which can do things for you. Not really compute things, but do stuff for you. I think this is nicely related to other personal apps Sandstorm is targeting.

[mitar]

@kentonv: Any suggestions on how would you implement this, if you would have time? Maybe I will have time to look into this. Or is this going into too much internals?

[kentonv]

@mitar The issue is not with CPU usage but RAM usage. Unfortunately, most apps use about the same amount of RAM whether or not they are currently in-use.

Implementing accounting and quotas for RAM is complicated low-level work involving cgroups and other kernel features, and is probably the kind of thing that needs to go into the non-open-source parts of Oasis.

[mitar]

Yea, having whole app up just for some background task is probably overkill. I am not even sure if quotas are a solution here. Or maybe some addition to the architecture allowing background jobs to run somehow without bringing up the whole grain. I would not be wanting to pay the whole "app consumption" for 24 hours a day just so that it can woke up for 5 minutes every 24 hours.

[kentonv]

Right, the right answer here is to have an API for scheduling events, so that Sandstorm starts up the grain when needed, so that it doesn't have to run 24/7.

[mitar]

Or that some subset of the app only starts up. For example, for Meteor, no need to to load most of it. :-)

[mitar]

But isn't grains not being on all the time also an issue for external API requests? Which would have to wait for the grain to come up?

[ocdtrekkie]

@mitar Nope. Sandstorm starts up the grain to answer an API request. They do wait, currently.

[mitar]

What about to handle incoming e-mail?

[zarvox]

Inbound email also starts up the grain: https://github.com/sandstorm-io/sandstorm/blob/master/shell/server/drivers/mail.js#L131

From the outside looking in, every external interaction with a grain can lazy-start the grain, because grains have no direct connection to the outside world, and everything is proxied through Sandstorm.

[mitar]

Hm, then maybe the easiest thing would be to have something like starting the grain from outside at regular time. Imagine that Sandstorm would call API every day, so that grain can do something. So if Sandstorm is starting grain for mail and API, then it can also start at regular interval. And then free plan would get Sandstorm to start it once a day, but if somebody wants it more regularly, they can pay. Or something like that.

[juanjux]

A workaround I use for the Tiny RSS problem is this, share a public link to the grain and then add this to your crontab:

*/30 * * * * /usr/bin/nice -n 19 timeout 60s /usr/bin/google-chrome-unstable --headless --remote-debugging-port=9999 --disable-gpu
     https://[shared_grain_url] >/dev/null 2>&1    

Not the nicest solution, but it works. It requires Chrome 57+ (currently unstable) because of the --headless parameter. Other option would be to use Phantom.js or something like that that can load a page and wait for the AJAX and JS content to load, but this is simpler.

[kentonv]

Instead of using Chrome, you can curl the API endpoint. Click the key icon in the topbar and generate a webkey, which has the form <url>#<token>. Then periodically do curl like:

curl -H "Authorization: Bearer <token>" <url>

[juanjux]

@kentonv thank you! Yes, this seems to work and it's much cleaner, I struggled a little before I noticed that the URL is the URL of the web key and not the one of the grain.

[zenhack]

I suspect now that powerbox-between-apps is implemented, it would be possible to write a "cron app" which itself gets a wake lock and can wake apps at requested times.

[kentonv]

@zenhack True, with the caveat that wake locks are not entirely reliable -- if the grain dies for any reason, it won't be started up again until the user opens it.

[mrdomino]

I'm taking a look at this now. I am tempted to add the scheduling API to SandstormApi directly. The first thought that comes to mind is to add methods for enumerating all tasks, adding a task, and deleting the task at a given index. I could also be convinced that returning a persistable handle for scheduled tasks is a good idea. Here's what I'm thinking now:

interface SandstormApi {
  # ...

  addScheduledWake @11 (schedule :WakeSchedule) -> (index :UInt64);
  listScheduledWakes @12 () -> (list :List(WakeSchedule));
  removeScheduledWake @13 (index :UInt64);

  struct WakeSchedule {
    tag @0 :Data;
    # Application-controlled tag.

    spec @1 :Spec;
    # Schedule specification TBD.
  }
}

@kentonv does this seem reasonable? Should I be going in a completely other direction, building a cron app or something?

EDIT: probably clearer to call them "wake"s than "task"s since they're just waking the whole grain up.

[zenhack]

Re: lack of restart, might make sense to have something where if a grain was holding a wake lock and the system goes down, the grain is restarted when the system comes back up.

Having "support grains" like that is also likely to make for a weird ux; needing to install cron and create a grain is something users shouldn't be asked to do.

That said, I worry about baking too much functionality into sandstorn core.

Here's a thought: for now, make the new wakeup api something that is requested via the powerbox. Sandstorm can fulfill the request itself, but this hides the fact that this functionally is "different" from apps.

Longer term, it might be nice from an architectural/implementation standpoint to have "system grains" that don't show up in the grain list, but exist by default and can fulfill powerbox requsts. We could down the road add a "cron" system grain. This gives us a modular/extensible way of adding apis without increasing architectural complexity.

Thoughts?

[ocdtrekkie]

I also have a hard time picturing having to install a Cron app (or even having one preinstalled... most users don't know what Cron is). And if someone accidentally deleted their Cron grain, all their other apps would stop working correctly. This might end up getting really confusing.

But I like your thoughts on having "system grains" from the sense that it'd be easy to rip and replace such a thing separately from the main Sandstorm install. But grains are, by definition, containerized instances of an app, and I'm not sure whether that makes sense for something like Cron. It shouldn't need to be heavily containerized, since it will start events in grains' own containers. And it's important for the server to be able to work with all of the Cron requests from all users and grains, so that it can schedule them intelligently to avoid resource contention.

The biggest challenge is that whether or not it is initially included, some UI is going to need to exist somewhere for users to see what grains are using the scheduler, and revoke their scheduling behavior. It needs to be possible at the user level, since they may be on a server with quota'd resources like Oasis they have to manage. (And for personal/family servers, it's likely that an admin needs to be able to track these down as well.) For the admin, this easily makes sense to be in the admin panel, much like the IpNetwork capabilities are. But I can't really see where the UI should be for users. Maybe @neynah has a thought. It may be possible to put off a user-facing UI, since Oasis doesn't currently regulate Compute Units, and if resources gets problematic, presumably Kenton would look into it.

[ocdtrekkie]

Maybe once Compute Unit tracking exists, a user facing list of apps resource usage would help point users to the grains with malfeasant scheduling behavior. Kinda like the battery usage chart on a phone.

[kentonv]

So I had a really old proposal already in a git branch and I cleaned it up: #2885

[kentonv]

Huh, I guess my API is more capability-ish whereas @mrdomino's is more REST-y.

I wonder if my desire to express everything in terms of objects and capabilities is overcomplicated. (Though usually when I think about this for awhile I remember a bunch of advantages to the capability approach...)

[zenhack]

IMO the complexities in the sandstorm API have nothing to do with the capability approach. The API you proposed is fairly similar to what I had in mind.

Here's my take, with an eye towards simplification:

Keep the Runnable interface, and then add something like:

@0xc5a4156fdf272f81;

interface Scheduler {
    schedule @0 (when :Util.DateInNs, callback :Util.Runnable,
        slack :Util.DurationInNs = 0, period :Util.DurationInNs = 0)
        -> (handle :Util.Handle);
}

Which apps would request via the powerbox. This keeps the list of methods for SandstormApi from getting too cluttered (I worry about it becoming a God Object), and also gives some flexibility re: implementation.

Instead of creating a new enum type for the period, just pass in a numeric duration. Zero denotes a one-shot event, so you don't need two different methods. Get rid of minimumSchedulingSlack too; it can be replaced by a fairly simple statement along the lines of "Sandstorm is not a real-time operating system."

I have a bit of nagging dislike of the slack parameter. I see the use, but the amount of verbiage that it's adding to the docs bugs me. Again, I wonder if we can just kill it entirely, and replace it with a generic "the timing isn't that precise" (maybe with the gory details in the next paragraph). I'm not sure about this one.

I'll also just state that there's not a great need for expressive ways to state schedules; apps can always just add a new event when they wake up if they need to do something fancy. So I don't think @mrdomino's Spec type should be able to express more than the simple oneshot or periodic events in kenton's proposal.

[zenhack]

The adaptive nature of slack actually strikes me as very likely to cause surprises. If you schedule something a month out that's going to be a slack of 3-4 days. Especially if a developer has specified 3:00 am or such for some task for which the user probably shouldn't be around, and then not done the math to realize how wide the margin is, this is likely to cause confusion. Similar if someone tries to use it to schedule something to happen on a particular holiday.

Yes, developers should read the docs carefully and think these things through, but having an API whose default behavior may to cause weird and infrequent timing bugs strikes me as something to be avoided.

Off the top of my head I can think of a couple approaches that may result in more predictable behavior

1. Get rid of the parameter entirely, and do a best-effort attempt to do things in a timely manner. Docs should point out that this isn't a real-time scheduler.
2. Don't make the parameter optional; ask the developer to specify a range of acceptable times.

I'm still not sure how I feel about slack.

[mrdomino]

I could easily be convinced to extract the Scheduler interface and make it requestable via Powerbox. I'm not going to do it yet because it makes the implementation harder for me and I want to get something working to iterate from. We could easily spend weeks bikeshedding the API.

I don't have strong feelings about slack. I agree that the default behavior is weird on long-ish time intervals. No obvious solution comes to mind. Maybe one will while I'm writing it.

FYI I have only one more day of vacation in the current stint, but I intend to take another 3-5 days in a couple weeks to finish this.

[mrdomino]

FYI I'm going into silent retreat soon. I'll be back on the 23rd. I hope to finish up the implementation by the end of the month, but it's possible that I won't be able to take enough free time for it. I'll keep you posted. If anyone just flat-out does this while I'm away, no hard feelings.

[mrdomino]

Well, I've burned through my current round of vacation time on this. No luck yet, although I understand Sandstorm's internals much better now than I did previously. I'll be stopping work on this for now -- anyone else, please feel free to take it up, and I'm happy to be a resource for you. If time permits, I may continue to work on it, but no guarantees.

The current point of confusion I have is where the scheduler should live. My best guess given my current understanding is that it should be on the frontend, since that's the place that has access to both persistent storage (in mongo) and continuous execution, but that seems to indicate that it should be in JavaScript. Not that big a deal, but kind of strange. Any thoughts on that @kentonv?

[kentonv]

Yeah, the scheduler would definitely need to be in the frontend, in Node.js Javascript. The name "frontend" is kind of misleading -- this is where all Sandstorm "business logic" lives. Look at the way backgrounding (aka "ongoing notifications") is done currently -- scheduled tasks will be somewhat different but you'll get an idea of what the codebase looks like.

Sorry that I haven't been readily available to answer questions. I should probably install an IRC client at work...

Nice that you're learning about the internals, though! Always good to have more people who know their way around the codebase. :)

[ocdtrekkie]

The Cron API has landed.