search

jhuangtw / xg2xg

by ex-googlers, for ex-googlers - a lookup table of similar tech & services
14.63k stars 1.04k forks source link

Google Workflow/TaskMaster equivalent #135

Open spaskob opened 2 years ago

spaskob commented 2 years ago

This was a very simple and powerful framework for building continuous pipelines at Google. I am not able to find anything remotely close to it. Any suggestions?

mkmik commented 6 months ago

A few years ago I started a quick&dirty tool (https://github.com/mkmik/masstasker) that is inspired by TaskMaster and I recently updated it to handle some practical workloads I have at $dayjob.

It's not a full reimpl (also because I don't really remember much of the details, certainly not the impl itself) but it's inspired by my memories of what I liked about it:

  1. immutable tasks
  2. protobufs for user-defined task bodies
  3. simple gRPC API you can manually use to operate on tasks
  4. lightweight and simple to setup
  5. not too high level, not too low level