HKube is a cloud-native open source framework to run distributed pipeline of algorithms built on Kubernetes.
HKube optimally utilizing pipeline's resources, based on user priorities and heuristics.
Distributed pipeline of algorithms
Language Agnostic - As a container based framework designed to facilitate the use of any language for your algorithm.
Batch Algorithms - Run algorithms as a batch - instances of the same algorithm in order to accelerate the running time.
Optimize Hardware Utilization
Build API - Just upload your code, you don't have to worry about building containers and integrating them with HKube API.
Cluster Debugging
Jupyter Integration - Scale your jupyter running tasks Jupyter with hkube.
HKube runs on top of Kubernetes so in order to run HKube we have to install it's prerequisites.
Kubernetes - Install Kubernetes or Minikube or microk8s.
Helm - HKube installation uses Helm, follow the installation guide.
Add the HKube Helm repository to helm
:
helm repo add hkube http://hkube.io/helm/
Configure a docker registry for builds
Create a values.yaml
file for custom helm values
build_secret:
# pull secret is only needed if docker hub is not accessible
pull:
registry: ''
namespace: ''
username: ''
password: ''
# enter your docker hub / other registry credentials
push:
registry: '' # can be left empty for docker hub
namespace: '' # registry namespace - usually your username
username: ''
password: ''
Install HKube chart
helm install hkube/hkube -f ./values.yaml --name my-release
This command installs HKube in a minimal configuration for development. Check production-deployment.
There are three ways to communicate with HKube: Dashboard, REST API and CLI.
Dashboard is a web-based HKube user interface. Dashboard supports every functionality HKube has to offer.
HKube exposes it's functionality with REST API.
{yourDomain}/hkube/api-server/swagger-ui
hkubectl
is HKube command line tool.
hkubectl [type] [command] [name]
# More information
hkubectl --help
Download hkubectl
latest version.
curl -Lo hkubectl https://github.com/kube-HPC/hkubectl/releases/latest/download/hkubectl-linux \
&& chmod +x hkubectl \
&& sudo mv hkubectl /usr/local/bin/
For mac replace with hkubectl-macos
For Windows download hkubectl-win.exe
Config hkubectl
with your running Kubernetes.
# Config
hkubectl config set endpoint ${KUBERNETES-MASTER-IP}
hkubectl config set rejectUnauthorized false
Make sure
kubectl
is configured to your cluster.HKube requires that certain pods will run in privileged security permissions, consult your Kubernetes installation to see how it's done.
We want to solve the next problem with given input and a desired output:
N
, k
.M
so: For example: N=5
, k=2
will result:
We will solve the problem by running a distributed pipeline of three algorithms: Range, Multiply and Reduce.
Creates an array of length N
.
N = 5
5 -> [1,2,3,4,5]
Multiples the received data from Range Algorithm
by k
.
k = 2
[1,2,3,4,5] * (2) -> [2,4,6,8,10]
The algorithm will wait until all the instances of the Multiply Algorithm
will finish then it will summarize the received data together .
[2,4,6,8,10] -> 30
We will implement the algorithms using various languages and construct a pipeline from them using HKube.
The pipeline descriptor is a JSON object which describes and defines the links between the nodes by defining the dependencies between them.
{
"name": "numbers",
"nodes": [
{
"nodeName": "Range",
"algorithmName": "range",
"input": ["@flowInput.data"]
},
{
"nodeName": "Multiply",
"algorithmName": "multiply",
"input": ["#@Range", "@flowInput.mul"]
},
{
"nodeName": "Reduce",
"algorithmName": "reduce",
"input": ["@Multiply"]
}
],
"flowInput": {
"data": 5,
"mul": 2
}
}
Note the
flowInput
:data
= N = 5,mul
= k = 2
HKube allows special signs in nodes input
for defining the pipeline execution flow.
In our case we used:
(@) — References input parameters for the algorithm.
(#) — Execute nodes in parallel and reduce the results into single node.
(#@) — By combining #
and @
we can create a batch processing on node results.
We created a pipeline name numbers
.
"name":"numbers"
The pipeline is defined by three nodes.
"nodes":[
{
"nodeName":"Range",
"algorithmName":"range",
"input":["@flowInput.data"]
},
{
"nodeName":"Multiply",
"algorithmName":"multiply",
"input":["#@Range","@flowInput.mul"]
},
{
"nodeName":"Reduce",
"algorithmName":"reduce",
"input":["@Multiply"]
},
]
In HKube, the linkage between the nodes is done by defining the algorithm inputs. Multiply
will be run after Range
algorithm because of the input dependency between them.
Keep in mind that HKube will transport the results between the nodes automatically for doing it HKube currently support two different types of transportation layers object storage and files system.
The flowInput
is the place to define the Pipeline inputs:
"flowInput":{
"data":5,
"mul":2
}
In our case we used Numeric Type but it can be any JSON type (Object
, String
etc).
There are more features that can be defined from the descriptor file.
"webhooks": {
"progress": "http://my-url-to-progress",
"result": "http://my-url-to-result"
},
"priority": 3,
"triggers":
{
"pipelines":[],
"cron":{}
}
"options":{
"batchTolerance": 80,
"concurrentPipelines": 2,
"ttl": 3600,
"progressVerbosityLevel":"info"
}
webhooks - There are two types of webhooks, progress and result.
You can also fetch the same data from the REST API.
{jobId}/api/v1/exec/status
{jobId}/api/v1/exec/results
priority - HKube support five level of priorities, five is the highest. Those priorities with the metrics that HKube gathered helps to decide which algorithms should be run first.
triggers - There are two types of triggers that HKube currently support cron
and pipeline
.
Check cron editor in order to construct your cron.
options - There are other more options that can be configured:
trace
debug
info
warn
error
critical
.The pipeline is built from algorithms which containerized with docker.
There are two ways to integrate your algorithm into HKube:
python
,javaScript
, java
and .NET core
.We will create the algorithms to solve the problem, HKube currently support two languages for auto build Python and JavaScript.
Important notes:
- Installing dependencies During the container build, HKube will search for the requirement.txt file and will try to install the packages from the pip package manager.
- Advanced Operations HKube can build the algorithm only by implementing start function but for advanced operation such as one time initiation and gracefully stopping you have to implement two other functions
init
andstop
.
def start(args):
print('algorithm: range start')
input = args['input'][0]
array = list(range(input))
return array
The start method calls with the args parameter, the inputs to the algorithm will appear in the input
property.
The input
property is an array, so you would like to take the first argument ("input":["@flowInput.data"]
as you can see we placed data
as the first argument)
def start(args):
print('algorithm: multiply start')
input = args['input'][0]
mul = args['input'][1]
return input * mul
We sent two parameters "input":["#@Range","@flowInput.mul"]
, the first one is the output from Range
that sent an array of numbers, but because we using batch sign (#) each multiply algorithm will get one item from the array, the second parameter we passing is the mul
parameter from the flowInput
object.
module.exports.start = args => {
console.log('algorithm: reduce start');
const input = args.input[0];
return input.reduce((acc, cur) => acc + cur);
};
We placed ["@Multiply"]
in the input parameter, HKube will collect all the data from the multiply algorithm and will sent it as an array in the first input parameter.
After we created the algorithms, we will integrate them with the CLI.
Can be done also through the Dashboard.
Create a yaml
(or JSON
) that defines the algorithm:
# range.yml
name: range
env: python # can be python or javascript
resources:
cpu: 0.5
gpu: 1 # if not needed just remove it from the file
mem: 512Mi
code:
path: /path-to-algorithm/range.tar.gz
entryPoint: main.py
Add it with the CLI:
hkubectl algorithm apply --f range.yml
Keep in mind we have to do it for each one of the algorithms.
Create a yaml
(or JSON
) that defines the pipeline:
# number.yml
name: numbers
nodes:
- nodeName: Range
algorithmName: range
input:
- '@flowInput.data'
- nodeName: Multiply
algorithmName: multiply
input:
- '#@Range'
- '@flowInput.mul'
- nodeName: Reduce
algorithmName: reduce
input:
- '@Multiply'
flowInput:
data: 5
mul: 2
For running our pipeline as raw-data:
hkubectl exec raw --f numbers.yml
First we store the pipeline:
hkubectl pipeline store --f numbers.yml
Then you can execute it (if flowInput
available)
# flowInput stored
hkubectl exec stored numbers
For executing the pipeline with other input, create yaml
(or JSON
) file with flowInput
key:
# otherFlowInput.yml
flowInput:
data: 500
mul: 200
Then you can executed it by pipeline name
:
# Executes pipeline "numbers" with data=500, mul=200
hkubectl exec stored numbers --f otherFlowInput.yml
As a result of executing pipeline, HKube will return a jobId
.
# Job ID returned after execution.
result:
jobId: numbers:a56c97cb-5d62-4990-817c-04a8b0448b7c.numbers
This is a unique identifier helps to query this specific pipeline execution.
Stop pipeline execution:
hkubectl exec stop <jobId> [reason]
Track pipeline status:
hkubectl exec status <jobId>
Track pipeline result:
hkubectl exec result <jobId>