search

Gi-z / CSIKit

Python CSI processing and visualisation tools for Atheros, Intel, Nexmon, ESP32, FeitCSI, and PicoScenes (USRP, etc) formats.
https://Gi-z.github.io/CSIKit/
MIT License
206 stars 51 forks source link
atheros csi csi-data csi-extraction-suite csi-parsing esp32 intel numpy picoscenes python visualisation wifi-hardware

readme

CSIKit Build Downloads

Tools for extracting Channel State Information from formats produced by a range of WiFi hardware/drivers, written in Python with numpy.

Python 3.8+ required.

Documentation updates are incoming.

Don't have your own CSI data? Check out CSI-Data for a collection of public CSI datasets.

CSIKit Command Line Example

Description

CSIKit is a framework aimed at assisting data scientists, researchers, and programmers with performing experiments and tests using CSI-capable WiFi hardware.

While the various public extraction toolkits do include scripts for parsing CSI data from their specific formats, these are largely written for MATLAB. Given the increasing focus on using deep learning in CSI experimentation, Python tools for parsing and processing CSI data may be more desirable for some. This is aimed at improving the accessibility of CSI research for those who may be interested in the area but have little experience with network engineering.

As is usually the case with research-oriented software, documentation is in-progress.

CSIKit provides a command line tool for parsing, processing, converting, and visualisation of CSI data, as well as libraries for use in other Python applications such as those working with Tensorflow, PyTorch, etc.

csikit [OPTIONS] file[.pcap/.dat/.csv/.csi]

Installation

CSIKit can be installed via pip or used directly from source.

pip install csikit

Example

Command Line

csikit log.all_csi.6.7.6.dat
csikit --graph --graph_type all_subcarriers log.all_csi.6.7.6.dat
csikit --csv log.all_csi.6.7.6.dat

CSIKit library

Generic example:

from CSIKit.reader import get_reader

my_reader = get_reader("path/to/csi_file.dat/pcap")
csi_data = my_reader.read_file("path/to/my_csi_file.dat/pcap")

Hardware-specific (Intel IWL5300) example:

from CSIKit.reader import IWLBeamformReader

my_reader = IWLBeamformReader()
csi_data = my_reader.read_file("path/to/log.all_csi.6.7.6.dat")

Library

CSIKit exposes key components for use in other Python applications: Reader and csitools. As other sections of code become more refined, this will likely grow over time.

Note: This documentation is initial and brief. More will follow shortly.

A Reader is used to read a given CSI capture file and generate parsed traces and matrices. As each file format differs a significant amount, different readers are necessary for each piece of hardware. Using the get_reader method in CSIKit.reader, the correct reader can be automatically selected for a given file. Optionally, hardware-specific readers can be imported, as ATHBeamformReader, IWLBeamformReader, NEXBeamformReader, CSVBeamformReader, PicoScenesBemaformReader, FeitCSIBeamformReader.

Once instantiated, a Reader can be used to read a file using the read_file method. This method returns a CSIData object, which contain frames (CSIFrame objects), timestamps, and metadata from parsing (errors, device information, etc). Additional parsing options can be passed to read_file, including scaled (bool) to rescale CSI values from manufacturer's internal scaling.

CSIFrames will have a different type, based on the type of Reader used to parse them. All CSIFrames contain a csi_matrix (being the most significant part of the payload), however they also contain additional attributes for each frame. Some attributes may be common across the different frame types, however they can vary significantly from one type to another. Next steps for this library are aimed at further integrating these standards. The full list of attributes associated with each frame are listed below.

Using CSI Matrices

CSI matrices contain complex numbers representing the gain and phase of a signal at a given subcarrier. These are output in their complex form in CSIFrames, however csitools contains a function for retrieving CSI amplitude from them.

from CSIKit.reader import get_reader
from CSIKit.util import csitools

my_reader = get_reader("path/to/file.pcap")
csi_data = my_reader.read_file("path/to/file.pcap", scaled=True)
csi_matrix, no_frames, no_subcarriers = csitools.get_CSI(csi_data)

The returned tuple contains a modified matrix which contains CSI amplitudes in dBm, followed by the number of frames and subcarriers represented therein.

Once we have CSI amplitude data, we can also apply filters for preprocessing (as seen in many publications making use of CSI).

This example below loads a given CSI file, applies some basic preprocessing, and plots the data in a heatmap.

from CSIKit.filters.passband import lowpass
from CSIKit.filters.statistical import running_mean
from CSIKit.util.filters import hampel

from CSIKit.reader import get_reader
from CSIKit.tools.batch_graph import BatchGraph
from CSIKit.util import csitools

import numpy as np

my_reader = get_reader("/path/to/csi/file")
csi_data = my_reader.read_file("/path/to/csi/file", scaled=True)
csi_matrix, no_frames, no_subcarriers = csitools.get_CSI(csi_data, metric="amplitude")

# CSI matrix is now returned as (no_frames, no_subcarriers, no_rx_ant, no_tx_ant).
# First we'll select the first Rx/Tx antenna pairing.
csi_matrix_first = csi_matrix[:, :, 0, 0]
# Then we'll squeeze it to remove the singleton dimensions.
csi_matrix_squeezed = np.squeeze(csi_matrix_first)

# This example assumes CSI data is sampled at ~100Hz.
# In this example, we apply (sequentially):
#  - a lowpass filter to isolate frequencies below 10Hz (order = 5)
#  - a hampel filter to reduce high frequency noise (window size = 10, significance = 3)
#  - a running mean filter for smoothing (window size = 10)

for x in range(no_frames):
  csi_matrix_squeezed[x] = lowpass(csi_matrix_squeezed[x], 10, 100, 5)
  csi_matrix_squeezed[x] = hampel(csi_matrix_squeezed[x], 10, 3)
  csi_matrix_squeezed[x] = running_mean(csi_matrix_squeezed[x], 10)

BatchGraph.plot_heatmap(csi_matrix_squeezed, csi_data.timestamps)

ATHCSIFrame

Reference based on the Atheros CSI Tool User Guide.

IWLCSIFrame

NEXCSIFrame

This format is based on the modified version of nexmon_csi (credit mzakharo) for BCM43455c0 with support for RSSI and Frame Control. If using the regular version of nexmon_csi, these fields will not contain this data.

FeitCSIFrame

Format based on FeitCSI format documentation

Supported Hardware

Coming Soon

Proper Documentation

Mistakes and Tests

If anything is wrong, let me know. I want to know why, and fix it!

I'm a PhD student working on several sensor data-focussed experiments, a few of which involve using CSI. I'm am by no means an RF engineer or particularly experienced this area. I have done and are doing as much as I can to make sure that anything produced with this is accurate. To that end, there are MATLAB .mat files included in the tests folder which have been generated using IWLBeamformReader, NEXBeamformReader, and scipy's savemat method. There are also MATLAB scripts in the scripts folder which can be used to check the validity of the output from this tool. In my experience I have found these tools to produce identical output to the MATLAB scripts offered by the following developers. If this is not the case, let me know.

Further to that, if there are any assertions I have made within code comments or otherwise which are factually incorrect, again let me know. I want to learn as much about this area as I reasonably can.

Reference Links

License

The code in this project is licensed under MIT license. If you are using this codebase for any research or other projects, I would greatly appreciate if you could cite this repository or one of my papers.

a) "G. Forbes. CSIKit: Python CSI processing and visualisation tools for commercial off-the-shelf hardware. (2021). https://github.com/Gi-z/CSIKit."

b) "Forbes, G., Massie, S. and Craw, S., 2020, November. WiFi-based Human Activity Recognition using Raspberry Pi. In 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI) (pp. 722-730). IEEE."

  @electronic{csikit:gforbes,
      author = {Forbes, Glenn},
      title = {CSIKit: Python CSI processing and visualisation tools for commercial off-the-shelf hardware.},
      url = {https://github.com/Gi-z/CSIKit},
      year = {2021}
  }

  @inproceedings{forbes2020wifi,
    title={WiFi-based Human Activity Recognition using Raspberry Pi},
    author={Forbes, Glenn and Massie, Stewart and Craw, Susan},
    booktitle={2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI)},
    pages={722--730},
    year={2020},
    organization={IEEE}
  }