SwiftFusion

A differentiable sensor fusion library written in Swift.

Think factor graphs (à la GTSAM) coupled with deep learning (via Swift for TensorFlow).

This project is in an early phase, but feel free to explore! Subject to massive changes. :-)

Getting started

Installation

macOS

Download a Swift for TensorFlow macOS toolchain and follow the macOS installation instructions. Installing the latest development snapshot is recommended. Check here for the "required" version of Xcode for using nightly toolchains. Sometimes toolchains may also work with older Xcode minor versions than the one listed.

To use Swift for TensorFlow, add the toolchain to PATH:

export PATH="/Library/Developer/Toolchains/swift-tensorflow-RELEASE-0.12.xctoolchain/usr/bin/:$PATH"

Linux

Download a Swift for TensorFlow Linux toolchain and follow the Linux installation instructions. Installing the latest development snapshot is recommended.

NVIDIA Jetson

Download the Swift for TensorFlow NVIDIA Jetson toolchain and follow the Linux installation instructions.

Testing

swift test --enable-test-discovery

Benchmarking

swift run -c release -Xswiftc -cross-module-optimization SwiftFusionBenchmarks

Development

Xcode (macOS)

Xcode provides a great Swift development experience on macOS.

To open SwiftFusion in Xcode, run open Package.swift.

Visual Studio Code

To enable Swift autocomplete in Visual Studio Code, install the Maintained Swift Development Environment plugin, and set the following plugin settings:

"sde.languageServerMode": "sourcekit-lsp",
"sourcekit-lsp.serverPath": "<your toolchain path>/usr/bin/sourcekit-lsp",
"sourcekit-lsp.toolchainPath": "<your toolchain path>",
"swift.path.swift_driver_bin": "<your toolchain path>/usr/bin/swift",

The CodeLLDB plugin enables debugging within Visual Studio Code. You need to set the following setting:

"lldb.library": "/swift-tensorflow-toolchain/usr/lib/liblldb.so"

A sample launch.json file:

{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "lldb",
      "request": "launch",
      "name": "Debug",
      "program": "${workspaceFolder}/.build/x86_64-unknown-linux-gnu/debug/SwiftFusionPackageTests.xctest",
      "args": [
        "--enable-test-discovery"
      ],
      "cwd": "${workspaceFolder}"
    }
  ]
}

Code overview

The main code is in Sources/SwiftFusion, organized under subdirectories.

Core

The core abstractions of SwiftFusion:

Vector.swift: protocol that formalizes a Euclidean vector space with standard orthonormal basis, and defines a great number of default methods for it. Also defines the ScalarsInitializableVector and FixedSizeVector protocols, and a collection StandardBasis<V: Vector> for the orthogonal bases.
Manifold.swift: protocol that inherits from Differentiable that adds retract and localCoordinate, which convert between a Manifold structure and its LocalCoordinate.
LieGroup.swift: protocol that formalizes making a differentiable manifold into a Lie group with composition operator *. Also defines LieGroupCoordinate protocol.

Specific Vector-like data structures:

VectorN.swift: generated by VectorN.swift.gyb, implements small fixed-size vectors conforming to AdditiveArithmetic, Vector, and FixedSizeVector.
FixedSizeMatrix.swift: matrix whose dimensions are known at compile time. Conforms to Equatable, KeyPathIterable, CustomStringConvertible, AdditiveArithmetic, Differentiable, FixedSizeVector.

Additional utilities:

DataTypes.swift: for now, just implements LieGroup protocol for Vector5
Dictionary+Differentiable.swift: conform Dictionary to Differentiable.
MathUtil.swift: for now just pinv (pseudo-inverse), using Tensor.svd.
TensorVector.swift: a view of a Tensor that conforms to the Vector protocol.
TrappingDouble.swift: a wrapper for Double that traps instead of allowing NaN.
Tuple+Vector.swift: (undocumented)
TypeKeyedArrayBuffers.swift: related to storage of Values and Factors.

Inference

FactorGraph.swift: the main factor graph structure storing factors.
Factor.swift: defines the core factor protocol hierarchy Factor → VectorFactor → LinearizableFactor → GaussianFactor, as well as the VariableTuple and DifferentiableVariableTuple protocols and Tuple conformances.
GaussianFactorGraph.swift: A factor graph whose factors are all GaussianFactors.

A factor graph stores factors in a storage array of type [ObjectIdentifier: AnyFactorArrayBuffer]. ObjectIdentifier is a built-in Swift identifier type and AnyFactorArrayBuffer is defined in FactorsStorage.swift.

Optimizers

GradientDescent.swift: a basic gradient descent optimizer.
CGLS.swift: a conjugate gradient solver for least-squares problems.
LM.swift: a Levenberg-Marquardt nonlinear optimizer.

MCMC

RandomWalkMetropolis.swift: implements a Metropolis sampler modeled after its equivalent in TensorFlow Probability.

Geometry

This directory defines rotation and pose types.

We use the GTSAM naming scheme: Pose2, Rot2, Pose3, and Rot3.

Pose2.swift: a 2D pose.
Rot2.swift: a 2D rotation.
Pose3.swift: a 3D pose.
Rot3.swift: a 3D rotation.

2D and 3D points are simply represented as Vector2 and Vector3.

Image

ArrayImage.swift: Differentiable multi-channel image stored as [Double]. Has a differentiable tensor method and update function.
OrientedBoundingBox.swift: a rectangular region of an image, not necessarily axis-aligned.
Patch.swift: a differentiable patch method for ArrayImage, that returns a resampled image for the given OrientedBoundingBox.

Datasets

This directory primarily contains support for reading G2O files.

LICENSE

Copyright 2020 The SwiftFusion Team

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

borglab / SwiftFusion

readme