search

Factual / skuld

Distributed task tracking system.
300 stars 13 forks source link

Task start time conditions #105

Open eric opened 10 years ago

eric commented 10 years ago

Tasks should be able to have a specified time that they are not started until. Tasks will not be eligible to claim until the time has passed.

eric commented 10 years ago

@aphyr: what format would you suggest specifying time in and what call should we use to get the current time?

aphyr commented 10 years ago

Uh, this opens up a whole nest of issues, including that people will think the option actually means something. I'd avoid implementing this if at all possible.

eric commented 10 years ago

I've ran into all sorts of reasons why I've wanted to put off running a task for some period of time.

What about something that says, "This can't be claimed until X seconds has passed"?

It would allow for doing backoff, but wouldn't be tied to a client clock.

aphyr commented 10 years ago

Skuld's design assumptions allow for clock skew on the order of minutes to hours. Second-scale timeouts are almost definitely not feasible.

eric commented 10 years ago

The problem I have that I was hoping to solve with this is cases where I look at a task and see that some other resource that this task depends on isn't available right now, so I don't need to try to run this task again for another X minutes.

As it is with a simple message queue (like what I use today), my only option is to throw it back into the end of the queue, which means that it will continue to see the same tasks over and over again while they're requeued until such a time that the other resource is ready again.

aphyr commented 10 years ago

You can use linear-time as a realllly loose proxy for "global time", but it can and will drift by huge volumes, and has no relation to anything outside Skuld. Heck, it may not even be close to other nodes.

aphyr commented 10 years ago

The correct solution is task dependencies. There's a writeup of this stuff in the Factual wiki; not sure if I ported it to the github repo.

aphyr commented 10 years ago

Oh, or, if your consumer is the one that discovers the dependent service is down, just have the consumer stop processing records for a bit.

eric commented 10 years ago

I see #15 that mentions dependencies, but I was assuming that just meant that one task has to complete before another one does.