Coinstac headless client TBN

[rssk]

Task Description

While there are several 'headless' client iterations, this one will be geared toward cluster use, as well as UI controllability. It may be useful to investigate later how the other clients can or cannot be rolled into this one

Requirements

• The headless client needs to be a standalone, packaged using pkg, binary that takes a data location, or is maybe passed a data location, maps, and listens for runs.
• Needs to be compatible with HPC schedulers (slurm, LSF, etc), this typically means the above so it can run as a shell command that can be launched off. This will likely also mean using the pipeline's network drive (alternative input directory) feature as default to not conflict with any drive mounting schemes
• The client will use a normal user account, but have an api key that allows access that will be passed in either command line or envar or file
• The client needs to, once launched, immediately check the api for any runs associated with the account and run.
• If we associate this with a normal user account, we'll have to figure out how to designate a local pipeline or a 'headless' cluster one. This could be a settings toggle or otherwise, or maybe chosen by the owner?
• Updates from this local client to the respective user UI might be nice, this could be tricky with GQL subscriptions, at worst case maybe a 'starting, running, finished' (cleaning up pipeline updates would help this + tracking down why updates are slow often)
• Data mapping is an open question, how do we map local data vs the UI? Do we want to send column data to the UI? If possible that may not even make sense as the headless client wont run until the scheduler lets it. Either a user manually enters the columns, and that generated mapping is sent to the headless client, or it's all done statically/semi-statically on the headless client end. Eg: the UI columns are sent to the headless client, then we use similar map matching logic to map them automatically. This will fall apart on any other client needed info that is not a csv file, like a boolean to toggle some local behavior.
• Results dont particularly need to be sent back to the UI, the user has access to the cluster unlike a vault
• using multiple jobs in parallel for hard tasks is desired, but that'll be in the future
• We're assume the task/pipeline finishes in the job time, resuming is also a possible future task

[dylanmartin]

While there are several 'headless' client iterations, this one will be geared toward cluster use, as well as UI controllability. It may be useful to investigate later how the other clients can or cannot be rolled into this one

Lets define and name things really clearly and distinctly We've used the word "headless" a number of times for different things already. Define the interface/use case/requirements we are trying to fit. We can begin by setting very narrow aims for this package and along the way find opportunities to decouple and isolate atomic pieces that can be more broadly useful through composition.

The use-case we're serving can be described as: "Running COINSTAC pipelines on computing clusters with HPC schedulers."

Maybe we make a package named something like "coinstac-client-hpc"

[dylanmartin]

• Updates from this local client to the respective user UI might be nice, this could be tricky with GQL subscriptions, at worst case maybe a 'starting, running, finished' (cleaning up pipeline updates would help this + tracking down why updates are slow often)

Are there currently separate channels for status updates to get to the UI? GraphQL subscriptions are one way information gets to the UI. Is there currently another form of communication that comes from the local pipeline to the UI?

[dylanmartin]

What is different about using COINSTAC with HPC schedulers than in a normal desktop environment?

[dylanmartin]

Here are some requirements I'm confident are needed:

• The client will need to communicate with the central API server
• It will also need to communicate using MQTT for pipeline runs
• Mapping files and column names needs to happen somehow