Adding first draft of paper

[vsoch]

Hey @yarikoptic! Just to get you excited, here is the early draft of our paper. We of course can finish this up if/when the datalad/reproman parser is added. In the meantime, do you have ideas for other (possibly non Dartmouth work related executors) that are generally useful in the research workflow space that I might work on?

Signed-off-by: vsoch vsochat@stanford.edu

[yarikoptic]

I would've better had us finalize datalad/reproman to possibly make it fit those use cases nicely, and then extend on demand. To my recent tweet a follow up referred me to https://github.com/pipitone/qbatch which also some to address one of the aspects of reproman - running jobs on batch systems (if you really want more ;-))

[yarikoptic]

Ideally there should be no parsing but rather an interface to talk through

[vsoch]

Hmm, so queueme (at face value) is intended for the user, issuing a command on the command line, and not the "user developer." But here's an idea, perhaps instead of adding some executor to queueme here, you would import the Queue into datalad,define your custom executor / template, and then control a version of QueueMe from within the python of your software. The executors that are provided in the repo here would need to satisfy having a "scientific user" user case that derives from a command line command.

[vsoch]

In other words, if it cannot be triggered from a command line and is mostly an interaction to another python library API, queueme should be imported there and not the other way around. It wouldn't be logical to provide an executor here that isn't accessible by the user via command line.

[yarikoptic]

Yes! That is pretty much what I meant by having cli/python in the original secret issue title. Also then y possibly provide their own actions (pretty much restful callbacks) to be triggered from the dashboard, and possibly even instruct which columns to be displayed etc ;-) that is why I think it would be great for us to chat at some point soon and align with those early ideas. Sorry for being slow with the reviews

[vsoch]

hey @yarikoptic ! Good talk yesterday! I figured I could put some thoughts here about the Datalad executor, since I thought about it last night. So - my thinking is that we could have a group of executors built into datalad that are distinguished from those based on a ShellExecutor because they aren't interacted with from a command line. E.g.,:

executors/
   __init__.py
    shell.py        (--- shell executor
    slurm.py      (--- shell executor
    datalad.py   (--- not a shell executor!

And the difference would be that there is no import to the init.py to try to match some random command on the command line to it. Instead, it would be used intentionally, and called from datalad. E.g.,:

from qme.main.executors.datalad import DataladRunExecutor
from qme.main import Queue

result = <do some datalad run thing here?>
task = DataladRunExecutor(result)
queue = Queue()
queue.add_task(task)

Or something like that? You mentioned something about a callback - how are you suggesting that would work?

[vsoch]

@yarikoptic I guess you never followed up here - should I close the issue? Worth developing more? Thoughts?

[yarikoptic]

Up to you

[vsoch]

I think we'd need a compelling use case = I want to have one but I'm not sure that I do yet :laughing: