sjdv1982 / seamless

Seamless is a framework to set up reproducible computations (and visualizations) that respond to changes in cells. Cells contain the input data as well as the source code of the computations, and all cells can be edited interactively.
http://sjdv1982.github.io/seamless
Other
20 stars 6 forks source link

ContextPool executor to execute multiple workflows in parallel #230

Open sjdv1982 opened 10 months ago

sjdv1982 commented 10 months ago

Mixed-style Seamless is expected to be in demand, especially in case of compiled transformers, which must always be part of a workflow, in combination with imperative syntaxes.

For the 0.12 release, there is already a planned feature seamless-runner / seamless-parallel-runner in order to execute workflow graphs from command-line (preferably under bin/seamless).

In addition, it will be necessary to execute workflows imperatively in parallel, as an imperative alternative to stdlib.map . For now, call this a ContextPool in analogy to multiprocessing.Pool . Here is some experimental working code that needs generalization:


NCONTEXTS=3

for n in range(NCONTEXTS):
    print(n)
    ctx = Context()
    ctx.set_graph(build_rotamer_graph)
    tf = ctx.build_rotamers
    tf.debug.direct_print = True
    ctx.translate()
    contexts.append(ctx)

for conformer in range(len(conformers)):
    print(conformer)
    ctx = contexts[conformer % NCONTEXTS]
    ctx.compute()
    if conformer >= NCONTEXTS:
        tf = ctx.build_rotamers
        print(conformer, tf.result.checksum, tf.status)        
    ctx.random_rotations = random_rotations
    ctx.scalevec = tensors[conformer][1]
    ctx.hierarchy = pre_analyses[conformer]["hierarchy"]

for conformer in range(len(conformers)-NCONTEXTS, len(conformers)):
    print(conformer)
    ctx = contexts[conformer % NCONTEXTS]
    ctx.compute()
    tf = ctx.build_rotamers
    print(conformer, tf.result.checksum, tf.status)