+TITLE: Parachute: a zero knowledge backup system

• Installation

Running

+NAME: Basic installation of parachute in one command

+BEGIN_SRC shell

bash -x ./setup/install-parachute.bash

+END_SRC

with the appropiate permissions should enable all the following subclauses to be met automatically ** Historical Install instructions in detail if ~./setup/install-parachute.bash~ doesn't work *** Place chute.asd in ASDF3 system registry

+NAME: Configuration of ASDF3 system registry to find blimp-parachute

+BEGIN_SRC shell

bash -x ./chute/configure-asdf.bash

+END_SRC

*** Loading from ASDF

+BEGIN_SRC lisp

(asdf:load-system :chute)

+END_SRC

*** Testing from ASDF

+BEGIN_SRC lisp

(asdf:test-system :chute)

+END_SRC

*** Loading Quicklisp dependencies

1. From the shell, download the Quicklisp install shim.

   +BEGIN_SRC shell

   wget https://beta.quicklisp.org/quicklisp.lisp

   +END_SRC

2. From a Common Lisp implementation, execute the shim.

   +BEGIN_SRC lisp

   (load "~/quicklisp.lisp") (quicklisp-quickstart:install) (ql:add-to-init-file)

   +END_SRC

3. Install the Quicklisp packages needed by chute.

   +BEGIN_SRC lisp

   (load (asdf:system-relative-pathname (asdf:find-system :chute) "quicklisp-setup.lisp"))

   +END_SRC

• Dockerification To build an image on arm32 with Docker installed:

  +BEGIN_SRC shell

  bash make-arm32-dockerfile.bash sudo docker build --file Dockerfile-arm32 .

  +END_SRC

• REST API ** Current version 'v1'

  +NAME: Fundamental Chute REST API

  +BEGIN_EXAMPLE

  --> POST /chute/blob/ (index.json)
  <-- 201 Resource Created or [45]xx Error ("/new/uri/to/use")

  --> GET /chute/blob//index.json
  nil <-- 200 Original or 304 Not Modified or [45]xx Error (index.json) or nil

  --> PUT /chute/blob//0
  (application/octet-bytes) <-- 201 Ok (json) "true" or "false"

  --> GET /chute/blob//0 <-- 200 (json) SHA256 Hash

+END_EXAMPLE

Authentication to be handled via HTTP Headers submitted to spire.marina.io API.

** Basic transfer of opaque binary data

PUT with 'Byte-range' headers n bytes in m windows. Use timing of previous sessions to dynamically expand/contract number of bytes in each PUT request.

Each chunk of a blob is adressed as a URI of the form:

.../<domain>/<node>/<mount-path>/<timestamp>/<n-byte-chunks>/<mth-chunk>

MIME type for PUT content is "application/octet-stream".

• Platform ~parachute~ provides both a client and a server implementation. All nodes boot can potentially boot either.

A local management REST API is always available bound to ~tcp4:2002~ Client platform A Common Lisp implementation on ~x86_64~, ~arm32~, or ~arm64~. ** CCL Most efficient runtime on arm32 platform http://trac.clozure.com/ccl/wiki/PlatformNotes SBCL * ABCL * Server platform A Common Lisp implementation running on ~x86_64~. SBCL ~sbcl-1.3.1~ or later known to work CCL ~ccl-1.11~ ABCL ~abcl-1.5.0~

• What is to be DONE ** DONE BTRFS volume must have ".snapshot/" directory CLOSED: [2015-12-14 Mon 13:55]
  • CLOSING NOTE [2015-12-14 Mon 13:55] \ Should have been completed in the setup procedure. File issue if found to be otherwise. The CHUTE:SNAPSHOT command will create snapshots under a the specified volume (by default "/opt/cloudfleet/data") in a sub-directory named ".snapshot/", i.e. for the default in "/opt/cloudfleet/data/.snapshot/".

Therefore, the "/opt/cloudfleet/data/.snapshot/" must exist and be owned by root. Currently this is done once per node as part of installation when "setup/add-subvolume.bash" is executed. It should be made part of the setup done by setting up the encrypted volumes.

** DONE make a hard link to btrfs, setuid CLOSED: [2015-10-27 Tue 12:44]

• CLOSING NOTE [2015-10-27 Tue 12:44] \ Done as setup/make-suid-btfs.bash

** BTRFS/SEND *** DONE stdout/stderr mixing CLOSED: [2015-11-14 Sat 10:16]

• CLOSING NOTE [2015-11-14 Sat 10:16] \ Fixed by specifying separate error/output arguments to CCL:RUN-PROGAM SBCL/CCL returning different starts of output stream. CCL contains "At subvolume" which means it is mixing stdout/stderr?

*** DONE Use octet streams CLOSED: [2015-11-19 Thu 10:47]

• CLOSING NOTE [2015-11-19 Thu 10:47] \ Resolved by use of octets streams directly on MAKE-BLOB We should pass the streams we wish to read from, rather than having RUN-PROGAM construct them for us.

Unfortunately, this doesn't seem to work:

+BEGIN_SRC lisp

(ccl:run-program "/bin/ls" nil :wait nil :output (ironclad:make-octet-input-stream (make-array 16 :element-type '(unsigned-byte 8))) :element-type '(unsigned-byte 8))

+END_SRC

complaining about

+BEGIN_EXAMPLE

There is no applicable method for the generic function:

<STANDARD-GENERIC-FUNCTION STREAM-WRITE-VECTOR #x30200006518F>

when called with arguments: (#<IRONCLAD::OCTET-INPUT-STREAM #x3020023CA75D> "chute LICENSE parachute.org setup ...

+END_EXAMPLE

Fortunately, CCL:RUN-PROGRAM takes an :ELEMENT-TYPE keyword which allows us to get octet streams out of the subprocesses.

** DONE parse id for PUT uri from POST CLOSED: [2015-12-14 Mon 14:09]

• CLOSING NOTE [2015-12-14 Mon 14:09] \ Finished with proper implementation of POST followed by subsequent PUTs to the unspecified sub-uri namespace. TODO Read key from key device Propsoal: extract appropiate number of bytes from SHA256 of the LuKS key TODO Initialize nonce from random data Done. But needs to be verified via a test. ** TODO Read domain from /opt/cloudfleet/data/config/blimp-vars.sh

Best implementation: exec a bash process then read its environment table ** DONE MAKE-NEW-DIRECTORY CLOSED: [2015-12-13 Sun 10:40]

• CLOSING NOTE [2015-12-13 Sun 10:40] \ Use CL-FAD routines instead of ASDF. Fails until run from the REPL. Unsure what this entails. ** TODO Verify basic transfer Basic transfer of backups needs to be implemented completely and tested:

*** REST Transfer Implementation Initial mplementation completed. Mocks in place for many other systems.

*** Tests of transfer integrity

**** CHUTE.TEST::TRANSFER.BLOB.1
Transform a given file into a blob

**** CHUTE.TEST::TRANSFER.BLOB.2

Use results of BTRFS/SEND into a blob

** TODO Future interface for subaddressing components of a blob For resumable transfers

+NAME: REST for resumable transfers

+BEGIN_SRC

            PUT /<URI>/0/<chunk-bytes>/<nth-chunk>
            ->>   201 on success or [345]00 
              (json) "true" or "false"

            GET /<URI>/0/<chunk-bytes>/<nth-chunk>/hash/sha256  
            ->>   20x [345]xx (does 314 make sense?)
              (json) SHA256 Hash

+END_SRC

** (at first without byte ranges). Implementation use HTTP 'Byte-range' header to files attached/detatched via mmap().

TODO Tests for use cases TODO Metadata transcriptions * URI Scheme for previous link TODO Generalize filesytem snapshoting abstraction Implement ZFS.

• Notes ** Osicat failing to grovel https://github.com/osicat/osicat/pull/11/files
• Colophon

  +BEGIN_EXAMPLE

  mark@evenson.eu Created: 01-OCT-2015 Revised: <2017-11-29 Wed 16:18>

  +END_EXAMPLE