First pass of "save" functionality

[tavinathanson]

What would be the best way to support Bfx.save writing to gs:// URLs?

@smondet I see the gsutil_cp function; is there an existing function that knows to write either locally or via gsutil? How about private gs:// URLs?

[tavinathanson]

From @smondet:

@tavinathanson it wouldn't be hard to implement (just look at the implementation of Bfx.input), then I'm not sure; maybe better doing Bfx.save to a directory and then, separately (throught Ketrew and the end of the pipeline or not), do gsutil rsync of the whole thing

[tavinathanson]

Probably would be a separate PR then?

[ihodes]

@smondet @tavinathanson I think it's best we merge this in, then I can file some issues for

1. supporting writing to gs://
2. generating a csv manifest
3. generating the .json files alongside the saved products

Does that seem reasonable?

[tavinathanson]

@ihodes works for me if @smondet is cool with the code

[tavinathanson]

@ihodes would love to use this in master, thoughts on getting it in?

[smondet]

@ihodes Travis says that you didn't update the src/examples/:

module My_cluster = struct

  let max_processors = 42

  let work_dir = "/work/dir/"

  let results_dir = "/results/dir/"

  let datasets_home = "/datasets/"

  let machine =

    Biokepi.Setup.Build_machine.create "ssh://example.com/tmp/KT/"

end

File "./src/examples/edsl_extension_register_result.ml", line 269, characters 7-182:

Error: Signature mismatch:

       ...

       The value `results_dir' is required but not provided

[ihodes]

This is ready to merge, I think! /cc @smondet

[tavinathanson]

@ihodes @smondet would love to use this in master today, so I vote for addressing anything else in a follow-up.