:information_source: Overview

Introduction

InterLink aims to provide an abstraction for the execution of a Kubernetes pod on any remote resource capable of managing a Container execution lifecycle. We target to facilitate the development of provider specific plugins, so the resource providers can leverage the power of virtual kubelet without a black belt in kubernetes internals.

The project consists of two main components:

• A Kubernetes Virtual Node: based on the VirtualKubelet technology. Translating request for a kubernetes pod execution into a remote call to the interLink API server.
• The interLink API server: a modular and pluggable REST server where you can create your own Container manager plugin (called sidecars), or use the existing ones: remote docker execution on a remote host, singularity Container on a remote SLURM batch system. This repository aims to maintain the SLURM sidecar as a standalone plugin.

The project got inspired by the KNoC and Liqo projects, enhancing that with the implemention a generic API layer b/w the virtual kubelet component and the provider logic for the container lifecycle management.

:electron: Usage

:bangbang: Requirements

• Our Kubernetes Virtual Node and the interLink API server

• The Go programming language (to build binaries)

• Docker Engine (optional)

Note: if you want a quick start setup (using a Docker container), Go is not necessary

:warning: Pods Requirements :warning:

• It is very important for you to remember to set CPU and Memory Limits in your Pod/Deployment YAML, otherwise default resources will be applied; specifically, if you don't set a CPU limit, only 1 CPU will be used for each task, while if you don't set any Memory limit, only 1MB will be used for each task.

• Docker entrypoints are not supported by Singularity. This means you have to manually specify a command to be executed. If you don't, /bin/sh is assumed as the default one.

The following is a simple example of a Pod with a specified command and limits properly set:

    apiVersion: v1
    kind: Pod
    metadata:
    name: test-pod
    annotations:
        slurm-job.knoc.io/flags: "--job-name=test-pod-cfg -t 2800  --ntasks=8 --nodes=1 --mem-per-cpu=2000"
    spec:
    restartPolicy: Never
    containers:
    - image: docker://busybox:latest 
        command: ["echo"]
        args: ["hello world"]
        imagePullPolicy: Always
        name: "hello world"
        resources:
            limits: 
                memory: 100Mi
                cpu: 2

:fast_forward: Quick Start

Just run:

cd docker && docker compose up -d

This way, a docker container with a full SLURM environment will be started. You can find the used SLURM configuration in /docker/SlurmConfig.yaml. If you want to update the config after you already started it once, run:

docker compose up -d --build --force-recreate

So, the old container will be deleted, the image rebuilt and a new container with the updated config will be deployed.

:hammer: Building binaries

It is of course possible to use binaries as a standalone application. Just run

make all

and you will be able to find the built slurm-sd binary inside the bin directory. Before executing it, remember to check if the configuration file is correctly set according to your needs. You can find an example one under examples/config/SlurmConfig.yaml. Do not forget to set the SLURMCONFIGPATH environment variable to point to your config.

:gear: A SLURM config example

SidecarPort: "4000"
SbatchPath: "/usr/bin/sbatch"
ScancelPath: "/usr/bin/scancel"
SqueuePath: "/usr/bin/squeue"
CommandPrefix: ""
SingularityPrefix: ""
ExportPodData: true
DataRootFolder: ".local/interlink/jobs/"
Namespace: "vk"
Tsocks: false
TsocksPath: "$WORK/tsocks-1.8beta5+ds1/libtsocks.so"
TsocksLoginNode: "login01"
BashPath: /bin/bash
VerboseLogging: true
ErrorsOnlyLogging: false

:pencil2: Annotations

:gear: Explanation of the SLURM Config file

:wrench: Environment Variables list

Here's the complete list of every customizable environment variable. When specified, it overwrites the listed key within the SLURM config file.