Local run of pipeline/algorithm

[yehiyam]

Should have an option to run on my machine but record the run in hkube, or on any other machine for that matter.

Run should be logged in hkube as-if it was run in a computer managed by hkube. for example - have an option to do something like hkubectl exec --local . To run on the local machine.

While running, the output folder and tensorboard folder would be watched and uploaded, as well as the logs.

This is very important in order to be able to develop the single algorithm on local machine first, and then integrate it to a pipeline. This way I can use any compute I have (e.g. a large compute cluster not part of hkube) for training, and record all experiments, and then run the same algorithm on hkube for inference.

copied from (5) in #973

[yehiyam]

Run one algorithm locally from IDE

1. Run hkubectl localRun start This starts a WS server (on port 3000) that will proxy all messages to and from the algorithm. In addition it will: a. make sure there is an algorithm (in debug mode) by that name in the cluster b. If the user requested to immediately execute the algorithm, will send /exec/algorithm command to the cluster

2. Run the algorithm in LocalRun mode:

   if __name__ == "__main__":
   Algorunner.RunLocal(name='fooalg1', start=start, execution={'input':[{'pipe':'true'}]}, single_run=True) 

   options:

   • name: the name of the algorithm, so the cli can create/verify the debug algorithm in the cluster
   • start: the start method
   • execution: dict with optional input for the algorithm
   • single_run: If True, will exit after the algorithm execution has finished

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.