search

MONROE-PROJECT / Scheduler

The MONROE scheduling server, and deployment daemon
https://www.monroe-project.eu/
0 stars 2 forks source link

readme

Marvin MONROE - a scheduling daemon

This package includes a relatively generic task scheduling daemon.

It is built to fulfill requirements of the European project MONROE, but should be suitable in other types of distributed network with a central controller.

Installation

python setup.py install

Copy and adjust the configuration files marvind.conf (on the node) or marvinctld.conf (on the scheduling server).

marvinctld

When started

All state changes are stored in a SQLite database on disk, and will be recovered on restart.

REST endpoints

routes = (
  '/version'          - protocol version number (GET)
  '/resources(|/.*)'   - node, type and status (GET, PUT)
  '/users(|/.*)'       - users (GET, PUT, POST, DELETE)
  '/experiments(|/.*)' - task definitions (GET, POST, DELETE)
  '/schedules(|/.*)'   - scheduled tasks (GET, PUT)
  '/backend(/.*)'     - backend actions (various)
)

Terminology:

Access levels:

'#' is a placeholder for a node id, task id, user id, iccid or scheduling id

for everyone:

for all authenticated clients:

only for users (role: user)

only for administrators (role: admin)

only for a node with the given id # (role: node)

To update the node status, send the schedules/# PUT request with the following fields:

Valid status codes are:

These status codes are final and cannot be overridden:

All status codes can be suffixed with ; and a reason (free text).

A valid traffic report is a JSON dictionary, and will overwrite the last traffic report for the same scheduling ID. To send a traffic report, provide the keys

The traffic report may contain arbitrary keys and values. It is planned to recognize the following keys for quota reimbursements:

Quota reimbursements are not implemented yet.

Experiment and scheduling parameters:

The following parameters are used to schedule an experiment:

ASAP scheduling: The scheduler selects the next available time slot after now.

LPQ scheduling: Low priority, queued. No time slot is assigned to the task. Instead, whenever the node is sending a heartbeat and has available capacity, this task will be immediately executed. LPQ scheduling will ignore recurrence parameters.

These are defined as scheduling options, interpreted by the scheduling server:

Options that are required to be known during deployment are passed to the node as deployment parameters.

The options parameter should be x-www-form-urlencoded, that is separated by ampersands and in the form key=value, or in the form of a JSON object.

Container parameters:

All options (including user provided keys that are not handled by the scheduler) are passed to the container in the /monroe/config file. Options prefixed with an underscore _ are hidden in the public user interface and API, and only passed to the container.

Authentication:

This is based on client certificates. The server is supposed to be run behind a HTTPS server (e.g. nginx) which will take care of verifying the certificate. That server will then have to set the header HTTP_VERIFIED to NONE or SUCCESS and the header HTTP_SSL_FINGERPRINT to the client certificate fingerprint

example nginx configuration:

proxy_set_header  VERIFIED         $ssl_client_verify;
proxy_set_header  SSL_FINGERPRINT  $ssl_client_fingerprint;

in order to use a web browser-based client, you also need to enable CORS headers, e.g.:

add_header 'Access-Control-Allow-Origin' "$http_origin" always;
add_header 'Access-Control-Allow-Credentials' 'true' always;
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS, PUT, DELETE' always;
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type' always;

if ($request_method = 'OPTIONS') {
    add_header 'Content-Length' 0 always;
    return 204;
}

To create an authority, server or client keys, follow this guide: http://dst.lbl.gov/~boverhof/openssl_certs.html

Configuration

An example configuration is provided in ./files/etc/marvinctld.conf

The files follow the YAML (http://yaml.org) syntax. The required keys are subject to change, but all keys should be mentioned in the example files.

marvind

When started

For every new task assigned:

Task status is reported back to the ctld.

Configuration

An example configuration is provided in ./files/etc/marvind.conf

EXAMPLE setup

On the scheduling server:

On the measurement node:

From anywhere:

POST /user Parameters: name, ssl, role

Creates user 'name' authenticated by the given ssl fingerprint and one of the valid roles (user, node, admin)

POST /experiment Parameters:

REST Return values: