neurons is a neural network library written from scratch in Rust. It provides a flexible and efficient way to build, train, and evaluate neural networks. The library is designed to be modular, allowing for easy customization of network architectures, activation functions, objective functions, and optimization techniques.

Jump to

• Features
• Quickstart
• The package
• Releases
• Progress
• Resources

Features

Modular design

• Ready-to-use dense, convolutional and maxpool layers.
• Inferred input shapes when adding layers.
• Easily specify activation functions, biases, and dropout.
• Customizable objective functions and optimization techniques.
• Feedback loops and -blocks for more advanced architectures.
• Skip connections with simple accumulation specification.
• Much more!

Fast

• Leveraging Rust's performance and parallelization capabilities.

Everything built from scratch

• Only dependencies are rayon and plotters.
  Where plotters only is used through some of the examples (thus optional).

Various examples showcasing the capabilities

• Located in the examples/ directory. With subdirectories for various tasks, showcasing the different architectures and techniques.

The package

The package is divided into separate modules, each containing different parts of the library, everything being connected through the network.rs module.

Core

tensor.rs

Describes the custom tensor struct and its operations.
A tensor is here divided into four main types:

• Single: One-dimensional data (Vec<_>).
• Double: Two-dimensional data (Vec<Vec<_>>).
• Triple: Three-dimensional data (Vec<Vec<Vec<_>>>).
• Quadruple: Four-dimensional data (Vec<Vec<Vec<Vec<_>>>>).

And further into two additional helper-types:

• Quintuple: Five-dimensional data (Vec<Vec<Vec<Vec<Vec<(usize, usize)>>>>>). Used to hold maxpool indices.
• Nested: A nested tensor (Vec<Tensor>). Used through feedback blocks.

Each shape following the same pattern of operations, but with increasing dimensions.
Thus, every tensor contains information about its shape and data.
The reason for wrapping the data in this way is to easily allow for dynamic shapes and types in the network.

random.rs

Functionality for random number generation.
Used when initializing the weights of the network.

network.rs

Describes the network struct and its operations.
The network contains a vector of layers, an optimizer, and an objective function.
The network is built layer by layer, and then trained using the learn function.
See quickstart or the examples/ directory for more information.

Layers

dense.rs

Describes the dense layer and its operations.

convolution.rs

Describes the convolutional layer and its operations.
If the input is a tensor of shape Single, the layer will automatically reshape it into a Triple tensor.

deconvolution.rs

Describes the deconvolutional layer and its operations.
If the input is a tensor of shape Single, the layer will automatically reshape it into a Triple tensor.

maxpool.rs

Describes the maxpool layer and its operations.
If the input is a tensor of shape Single, the layer will automatically reshape it into a Triple tensor.

feedback.rs

Describe the feedback block and its operations.

Functions

activation.rs

Contains all the possible activation functions to be used.

objective.rs

Contains all the possible objective functions to be used.

optimizer.rs

Contains all the possible optimization techniques to be used.

Examples

plot.rs

Contains the plotting functionality for the examples.

Quickstart

use neurons::{activation, network, objective, optimizer, tensor};

fn main() {

  // New feedforward network with input shape (1, 28, 28)
  let mut network = network::Network::new(tensor::Shape::Triple(1, 28, 28));

  // Convolution(filters, kernel, stride, padding, dilation, activation, Some(dropout))
  network.convolution(5, (3, 3), (1, 1), (1, 1), (1, 1), activation::Activation::ReLU, None);

  // Maxpool(kernel, stride)
  network.maxpool((2, 2), (2, 2));

  // Dense(outputs, activation, bias, Some(dropout))
  network.dense(100, activation::Activation::ReLU, false, None);

  // Dense(outputs, activation, bias, Some(dropout))
  network.dense(10, activation::Activation::Softmax, false, None);

  network.set_optimizer(optimizer::RMSprop::create(
      0.001,                     // Learning rate
      0.0,                       // Alpha
      1e-8,                      // Epsilon
      Some(0.01),                // Decay
      Some(0.01),                // Momentum
      true,                      // Centered
  ));
  network.set_objective(
      objective::Objective::MSE, // Objective function
      Some((-1f32, 1f32))        // Gradient clipping
  );

  println!("{}", network);       // Display the network

  let (x, y) = {  };             // Add your data here
  let validation = (
      x_val,                     // Validation data
      y_val,                     // Validation labels
      5                          // Stop if val loss decreases for 5 epochs
  );
  let batch = 32;                // Minibatch size
  let epochs = 100;              // Number of epochs
  let print = Some(10);          // Print every 10th epoch
  let (train_loss, val_loss, val_acc) = network.learn(x, y, validation, batch, epochs, print);
}

Releases

Progress

Resources

Sources

• backpropagation
• softmax
• momentum
• Adam
• AdamW
• RMSprop
• convolution 1
• convolution 2
• convolution 3
• skip connection
• feedback 1
• feedback 2
• transposed convolution

Tools used

• GitHub Copilot
• ChatGPT
• Mistral
• Claude

Inspiration

• candle
• rust-simple-nn