childmindresearch / bids2table

Efficiently index large-scale BIDS neuroimaging datasets and derivatives
https://childmindresearch.github.io/bids2table/
MIT License
13 stars 5 forks source link
arrow bids data-pipeline elt etl neuroimaging parquet

bids2table

Build Docs codecov Code style: black MIT License

bids2table is a library for efficiently indexing and querying large-scale BIDS neuroimaging datasets and derivatives. It aims to improve upon the efficiency of PyBIDS by leveraging modern data science tools.

bids2table represents a BIDS dataset index as a single table with columns for BIDS entities and file metadata. The index is constructed using Arrow and stored in Parquet format, a binary tabular file format optimized for efficient storage and retrieval.

Installation

A pre-release version of bids2table can be installed with

pip install bids2table

The latest development version can be installed with

pip install git+https://github.com/childmindresearch/bids2table.git

Documentation

Our documentation is here.

Example

import pandas as pd

from bids2table import bids2table

# Load in memory as pandas dataframe
df = bids2table("/path/to/dataset")

# Load in parallel and stream to disk as a Parquet dataset
df = bids2table("/path/to/dataset", persistent=True, workers=8)

See here for a more complete example.

Performance

bids2table significantly outperforms both PyBIDS and ancpBIDS in terms of indexing run time, index size on disk, and query run time.

Indexing performance

Indexing run time and index size on disk for the NKI Rockland Sample dataset. See the indexing benchmark for more details.

Index Num workers Run time (s) Index size (MB)
PyBIDS 1 1618 448
ancpBIDS 1 465 --
bids2table 1 402 4.02
bids2table 8 53.2 3.84
bids2table 64 10.7 4.82

Query performance

Query run times for the Chinese Color Nest Project dataset. See the query benchmark for more details.

Index Get subjects (ms) Get BOLD (ms) Query metadata (ms) Get morning scans (ms)
PyBIDS 1350 12.3 6.53 34.3
ancpBIDS 30.6 19.2 -- --
bids2table 0.046 0.346 0.312 0.352