Toolbox for MM-Fi Dataset

Introduction

MM-Fi is the first multi-modal non-intrusive 4D human pose estimation dataset with 27 daily or rehabilitation action categories for high-level wireless human sensing tasks. MM-Fi consists of over 320k synchronized frames of five modalities from 40 human subjects in four domains. The annotations include 2D/3D human pose keypoints, 3D position, 3D dense pose, and the category of action.

MM-Fi has been published in NeurIPS 2023 Datasets and Benchmarks Track [Paper].

For more details and demos about MMFi dataset, please refer to [Project Page].

Please download the dataset through [Google Drive] or _[Baidu Netdisk]_.

*The anonymized RGB images can be downloaded from _[Ali Cloud]_.

Quick Start for MMFi Toolbox

To get started, follow the instructions in this section. We will introduce the simple steps and how you can customize the configuration.

Step 1

Dependencies

Please make sure you have installed the following dependencies before using MMFi dataset.

• Python 3+ distribution
• Pytorch >= 1.1.0

Quick installation of depedencies (in one local or vitual environment)

pip install python torch torchvision pyyaml numpy scipy opencv-python

Step 2

Once the environment is built successfully, download the dataset;

After unziping all four parts, the dataset directory structure should be as follows.

Dataset Directory Structure

${DATASET_ROOT}
|-- E01
|   |-- S01
|   |   |-- A01
|   |   |   |-- rgb
|   |   |   |-- mmwave
|   |   |   |-- wifi-csi
|   |   |   |-- ...
|   |   |-- A02
|   |   |-- ...
|   |   |-- A27
|   |-- S02
|   |-- ...
|   |-- S10
|-- E02
|......
|-- E03
|......
|-- E04
|......

Step 3

Edit your code and configuration file (.yaml file) carefully before running. For details of the configuration, please check the keys description.

Here we just take the code snippets in the example.py for instance.

import yaml
import numpy as np
import torch

# Please add the downloaded mmfi directory into your python project. 
from mmfi import make_dataset, make_dataloader

dataset_root = '/data3/MMFi_Dataset'  # path will not be same in your server.
with open('config.yaml', 'r') as fd:  # change the .yaml file in your code.
    config = yaml.load(fd, Loader=yaml.FullLoader)

train_dataset, val_dataset = make_dataset(dataset_root, config)
rng_generator = torch.manual_seed(config['init_rand_seed'])
train_loader = make_dataloader(train_dataset, is_training=True, generator=rng_generator, **config['train_loader'])
val_loader = make_dataloader(val_dataset, is_training=False, generator=rng_generator, **config['validation_loader'])

# Coding

Step 4

Now you can start the implementation. For example, using the commands below:

cd your_project_dir
// make coding
python your_script_name.py path_to_dataset_dir your_config.yaml

Description of Keys in Configuration

modality

• Single modality

  Please use one of the followings:

  rgb, infra1, infra2, depth, lidar, mmwave, wifi-csi

  Note that every modality should be in lowercase.

  Currently, the raw images (rgb, infra1 and infra2) of subjects are not publicly available for the privacy concerns. Thus, we provide the 17 body keypoints extracted from images using the common ResNet-48 model.

• Multiple modalities:

  Please use | to connect different modalities.

  Note that space is not allowed in the connection. For example, wifi-csi|mmwave is OK, but wifi-csi | mmwave will not be accepted.

data_unit

• sequence

  The data generator will return data with sequence as the unit, e.g., each sample contains 297 frames.

• frame

  The data generator will return data with frame as the unit, e.g., each sample only has 1 frame.

protocol

This key defines how many activities could be enabled in your training/testing.

• protocol 1: Only the daily activities are enabled.
• protocol 2: Only the rehabilitation activities are enabled.
• protocol 3: All activities are enabled.

split

The train/test split of your code. There are already 3 splits which are used in our paper.

• manual_split: Please refer to the example in .yaml file and customize your own dataset split setting here (which subjects and actions are regarded as the testing data).
• split_to_use: Specify the split you want.

train_loader validation_loader

These two options define the parameters which are used to construct your dataloaders. We keep these two options open so that you could customize freely.

More Details about MMFi Dataset

Activities Included

MMFi dataset constains two types of actions: daily activities and rehabilitation activities.

Subjects and Environments

40 volunteers (11 females and 29 males) aging from 23 to 40 participated in the data collection of MMFi. We appreciate their kind assitance in the completion of this work!

In addition, the 40 volunteers were divided into 4 groups corresponding to 4 different environmental settings so that cross-domain research could be conducted for the WiFi sensing.

Action Segments

We have also extracted the action segments from the raw sequences, with relevant information stored into a .csv form, which can be referred to in the dataset directory.

Reference

Please cite the following paper if you find MMFi dataset and toolbox benefit your research. Thank you for your support!

@inproceedings{
    yang2023mm,
    title={MM-Fi: Multi-Modal Non-Intrusive 4D Human Dataset for Versatile Wireless Sensing},
    author={Yang, Jianfei and Huang, He and Zhou, Yunjiao and Chen, Xinyan and Xu, Yuecong and Yuan, Shenghai and Zou, Han and Lu, Chris Xiaoxuan and Xie, Lihua},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
    year={2023},
    url={https://openreview.net/forum?id=1uAsASS1th}
}