Datarray

[liquidev]

Name: Datarray

Author: liquidev

Datarray aims to make data oriented design easy, by abstracting away the common struct of arrays pattern that exhibits much better cache efficiency than the usual, object oriented-style array of structs. The creation of this library was inspired by this blog post.

Here's an example of how to use a datarray to manipulate data efficiently:

import std/random
import datarray

# declare a struct with fields that should be transformed into arrays
type
  Ant = object
    name: string
    color: string
    isWarrior: bool
    age: int32

# create and initialize the datarray with some data to operate on
var antColony: Datarray[1000, Ant]
randomize()
for ant in antColony:
  # `ant` is of a special type - `VarElement[Ant]`, which holds a pointer to the
  # datarray's memory, its length, and the relevant index of the element we're
  # operating on. there also exists an `Element[T]` for non-var datarrays that
  # prohibits assignment to fields.
  # the special . and .= operators can be used to access or modify the fields:
  ant.name = sample ["joe", "josh", "dave"]
  ant.color = sample ["red", "black", "brown"]
  ant.isWarrior = rand(1.0) < 0.5
  ant.age = int32 rand(1 .. 3)
  # {} and {}= can also be used if you prefer to avoid using experimental
  # language features, but to get stabilized, experimental features need testing
  # so it's your choice ;)

# query the data using the `select` macro
var numChosen = 0
for index, (age, color) in select antColony:
  # this macro invocation creates two templates inside of the loop,
  # `age` and `color`, which desugar to something like this:
  # antColony.ith(index, age)
  # to save typing.
  if age > 1 and color == "red":
    inc numChosen
echo "found ", numChosen, " red ants older than 1 year"

Because a datarray is quite a generic and flexible container, it can be used to implement a variety of different things, eg. an ECS framework, or a simple and fast database. A dynamically allocated variant, DynDatarray, is also available, if the size needs to be determined at runtime, but I'm yet to implement a dynamically resizable datarray as of writing this.

Benchmarks show that using an struct of arrays is about 2-3x faster than using an array of structs, so if you want to write fast programs that take full advantage of your CPU, start using datarrays now!

The library is still growing and lacks many useful features, so suggestions are welcome.