elixir-explorer / adbc

Apache Arrow ADBC bindings for Elixir
https://arrow.apache.org/adbc/
Apache License 2.0
50 stars 16 forks source link

feat: add support for decoding list views and large list views #89

Closed cocoa-xu closed 5 months ago

cocoa-xu commented 5 months ago

This PR adds support for decoding list views and large list views.

The result by default is represented in the following format, where data is a map consisting 4 key-value pairs:

iex> list_view = %Adbc.Column{
  name: "sample_list_view",
  type: :list_view,
  nullable: true,
  metadata: %{},
  data: %{
    validity: [true, false, true, true, true],
    offsets: [4, 7, 0, 0, 3],
    sizes: [3, 0, 4, 0, 2],
    values: %Adbc.Column{
      name: "item",
      type: :i32,
      nullable: false,
      metadata: %{},
      data: [0, -127, 127, 50, 12, -7, 25]
    }
  }
}

To convert from the compact form to lists, users can use

iex> Adbc.Column.list_view_to_list(list_view)
%Adbc.Column{
  name: "sample_list_view",
  type: :list,
  nullable: true,
  metadata: %{},
  data: [
    %Adbc.Column{
      name: "item",
      type: :i32,
      nullable: false,
      metadata: nil,
      data: [12, -7, 25]
    },
    nil,
    %Adbc.Column{
      name: "item",
      type: :i32,
      nullable: false,
      metadata: nil,
      data: [0, -127, 127, 50]
    },
    %Adbc.Column{
      name: "item",
      type: :i32,
      nullable: false,
      metadata: nil,
      data: []
    },
    %Adbc.Column{
      name: "item",
      type: :i32,
      nullable: false,
      metadata: nil,
      data: ~c"2\f"
    }
  ]
}

The example is taken from https://arrow.apache.org/docs/format/Columnar.html#listview-layout