How to use bistro as workqueue?

[yinlinzh]

Hi Developers,

I'm working on setting up distributed task scheduling system to achieve data migration from HDFS to local file system. The example use case is querying the namenode to get file list, then create and schedule the task for each individual file. In README, the example code shows that scheduling and executing one task which written in script. Here are some of my questions:

1. Is there any API list for other any language that integration with bistro?
2. How to use bistro as workqueue? In other words, after creating each task, how do I keep those created tasks? I noticed bistro can talk with MySQL, HBase and Postgres. Would you please provide more information about this?
3. My plan is to run "task agent" on each "worker", instead of shell script way, something there's daemon process running on worker and waiting for scheduling from scheduler. Would you please some examples?

Thanks in advance!

[snarkmaster]

1) What sort of integration are you looking for?

2) A few people have used Bistro as a work-queue in the past.

Are you trying to queue many sharded jobs, or to queue per-shard tasks for a single job? Can you give more detail?

The typical pattern for a queue of many jobs would be:

• Decide on some kind of database to keep the jobs. This can be as simple as a SQLite DB on the scheduler's host, or some kind of fancy distributed DB.
• Implement a ConfigLoader for that DB type supporting saveJob and deleteJob. There are some simple examples https://github.com/facebook/bistro/tree/master/bistro/config
• Enqueue jobs via the REST API, or (less preferred) by writing them directly to the underlying DB.

The typical pattern for a queue of many shards / tasks would be similar:

• Decide on a DB or data source -- note that tasks can be generated on the fly, if appropriate, instead of being stored anywhere. E.g. ls /some/path is a valid script_fetcher.
• Write a NodeFetcher, or use ScriptFetcher to have your scheduler run an external script to fetch the nodes. Code here: https://github.com/facebook/bistro/tree/master/bistro/nodes
• Enqueue shards / tasks by writing to the underlying data source.

3) I don't really understand your question. Can you try describing what you are trying to do, and how you want to do it in more detail? I assume you are aware of the binaries bistro_scheduler and bistro_worker, and how they interact? If not, you should try --help with both, and skim the protocol documentation:

• https://github.com/facebook/bistro/blob/master/bistro/if/README.worker_protocol
• (Advanced topic for later) https://github.com/facebook/bistro/blob/master/bistro/remote/README.worker_set_consensus

[snarkmaster]

@yinlinzh, please reopen if you would like to resume this discussion :)