frogmap - Githubissues

c-cube / frog-utils

[frozen] Scheduling and running jobs on a shared computer, then analyse their output

BSD 2-Clause "Simplified" License

5 stars 0 forks source link

frogmap #14

Closed c-cube closed 9 years ago

c-cube commented 9 years ago

make the frogmap tool+library:

Usage should be something like

frogmap -j 15 [options] -- cmd f1 f2 f3.... f1000

or

frogmap -j 15 -F <file listing problems> cmd

or

frogmap --resume <state file>

The state is stored in one file, that can be used to --resume a task that was killed/stopped.

[ ] file format to store the task state
- one json object at the top, with
- list of files/arguments to deal with
- the command
- the user, parameters, parallelism level
- one json object per line, with the result of the command on one file/arg
- remaining tasks = all tasks to deal with \ those whose result is already stored
[ ] tool that runs a given amount of tasks in parallel, using Lwt
[ ] provide a nice library to parse such files (to be used by frogfold)

c-cube commented 9 years ago

Done.