search

mne-tools / mne-bids

MNE-BIDS is a Python package that allows you to read and write BIDS-compatible datasets with the help of MNE-Python.
https://mne.tools/mne-bids/
BSD 3-Clause "New" or "Revised" License
135 stars 90 forks source link

Channels Weird Unit Error #394

Closed alexrockhill closed 4 years ago

alexrockhill commented 4 years ago

Using write_raw_bids, I get the following error for what looks to me to be correctly labelled channels. Perhaps something weird is going on with the ASCII representation of microvolts?

Screen Shot 2020-04-22 at 12 23 00 PM

name type units low_cutoff high_cutoff description sampling_frequency status Event TRIG n/a 0.0 499.70605526161086 Trigger 999.4121105232217 good LTPL1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LTPL2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LTPL3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LTPL4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LTPL5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAMY1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAMY2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAMY3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAMY4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAMY5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAMY6 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAMY7 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAHP1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAHP2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAHP3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAHP4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAHP5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAHP6 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAHP7 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LSTG1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LSTG2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LSTG3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LSTG4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LSTG5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LSTG6 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LSTG7 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAIN1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAIN2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAIN3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAIN4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LAIN5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LACG1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LACG4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LACG7 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LOFC1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LOFC2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LOFC3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LOFC4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LOFC5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LOFC6 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LOFC7 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LOFC8 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good LOFC9 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RTPL1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RTPL2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RTPL3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RTPL4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RTPL5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAMY1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAMY2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAMY3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAMY4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAMY5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAMY6 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAMY7 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAMY8 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAMY9 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAMY10 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAMY11 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAHP1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAHP2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAHP3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAHP4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAHP5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAHP6 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAHP7 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSTG1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSTG2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSTG3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSTG4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSTG5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSTG6 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSTG7 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSTG8 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSTG9 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSTG10 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSTG11 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSTG12 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSTG13 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RPLIN1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RPLIN2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RPLIN3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RPLIN4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RPLIN5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAIN1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAIN2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAIN3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAIN4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RAIN5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RACG1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RACG2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RACG3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RACG4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RACG5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RACG6 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RACG7 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good ROFC1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good ROFC2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good ROFC3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good ROFC4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good ROFC5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good ROFC6 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good ROFC7 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good ROFC8 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good ROFC9 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSLS1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSLS2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSLS3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSLS4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSLS5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSLS6 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSLS7 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSLS8 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RSLS9 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RALS1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RALS2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RALS3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RALS4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RALS5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RPLS1 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RPLS2 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RPLS3 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RPLS4 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good RPLS5 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good C127 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good C128 SEEG µV 0.0 499.70605526161086 StereoEEG 999.4121105232217 good

sappelhoff commented 4 years ago

potentially unrelated to your issue, but interesting BIDS news: Units are currently a mess, please see my summary here: https://github.com/bids-standard/bids-specification/pull/411#issuecomment-617036529

regarding your issue: it's unlikely an mne-bids issue, because we are getting the units directly from the raw using MNE: from mne.channels.channels import _unit2human, see write.py

jasmainak commented 4 years ago

@alexrockhill to begin to understand the issue, I would suggest sharing a minimal-reproducible code. Which variation of the micro sign do you have?

alexrockhill commented 4 years ago

The channels.tsv file can't be uploaded directly but I copy pasted it below the image. I'll see if it reproduces with sample data in an hour (in a lecture).

jasmainak commented 4 years ago

okay keep us posted! I think you should be able to reproduce. You just need to figure out which character it is that's causing the problem.

alexrockhill commented 4 years ago

Minimally reproducible example:

import os.path as op
import pytest
from glob import glob
from datetime import datetime, timezone
import platform
import shutil as sh
import json
from distutils.version import LooseVersion
from pathlib import Path

import numpy as np
from numpy.testing import assert_array_equal, assert_array_almost_equal

# This is here to handle mne-python <0.20
import warnings
with warnings.catch_warnings():
    warnings.filterwarnings(action='ignore',
                            message="can't resolve package",
                            category=ImportWarning)
    import mne

from mne.datasets import testing
from mne.utils import (_TempDir, run_subprocess, check_version,
                       requires_nibabel, requires_version)
from mne.io import anonymize_info
from mne.io.constants import FIFF
from mne.io.kit.kit import get_kit_info

from mne_bids import (write_raw_bids, read_raw_bids, make_bids_basename,
                      make_bids_folders, write_anat, make_dataset_description)
from mne_bids.write import (_stamp_to_dt, _get_anonymization_daysback,
                            get_anonymization_daysback)
from mne_bids.tsv_handler import _from_tsv, _to_tsv
from mne_bids.utils import _find_matching_sidecar
from mne_bids.pick import coil_type

base_path = op.join(op.dirname(mne.__file__), 'io')
subject_id = '01'
subject_id2 = '02'
session_id = '01'
run = '01'
acq = '01'
run2 = '02'
task = 'testing'

bids_basename = make_bids_basename(
    subject=subject_id, session=session_id, run=run, acquisition=acq,
    task=task)

bids_root = _TempDir()
data_path = op.join(testing.data_path(), 'EDF')
raw_fname = op.join(data_path, 'test_reduced.edf')

raw = mne.io.read_raw_edf(raw_fname)

write_raw_bids(raw, bids_basename, bids_root, overwrite=False)
jasmainak commented 4 years ago

I get:

NameError: name 'events_fname' is not defined
alexrockhill commented 4 years ago

Oops try now

alexrockhill commented 4 years ago

Weird that the _bids_validate doesn't throw an error. Maybe @sappelhoff knows how that works and if it's using a previous version (whereas https://bids-standard.github.io/bids-validator/ does give an error)

alexrockhill commented 4 years ago

Seems like there might be bigger issues but maybe a temporary work around would be to just convert the data into V for EEG

alexrockhill commented 4 years ago

Or would it be better to raise the issue with the validator as it looks like the microvolts not being handled properly are the root cause?

jasmainak commented 4 years ago

@alexrockhill this is not a minimal reproducible example :) This is better:

import os.path as op
import mne

from mne_bids import make_bids_basename, write_raw_bids

subject_id = '01'
subject_id2 = '02'
session_id = '01'
run = '01'
acq = '01'
run2 = '02'
task = 'testing'

bids_basename = make_bids_basename(
    subject=subject_id, session=session_id, run=run, acquisition=acq,
    task=task)

bids_root = './bids_dataset'
data_path = op.join(mne.datasets.testing.data_path(), 'EDF')
raw_fname = op.join(data_path, 'test_reduced.edf')

raw = mne.io.read_raw_edf(raw_fname)

write_raw_bids(raw, bids_basename, bids_root, overwrite=True)

have you given up on linting since you stopped contributing to mne-bids? :)

If you write to temp dir I cannot upload it to the validator because mac doesn't like random directories when uploading (at least I haven't found a way to just give a path)

I would say you should report the error in the package that does not conform to the specification. I haven't checked thoroughly -- would encourage you to dig into it and check which of the two packages is problematic. Thanks a lot!!

alexrockhill commented 4 years ago

On a mac if you press command+shift+G in the finder, you can paste a path in.

I was fending off my dog jumping on me wanting to go outside and play so my example was a bit rushed.

Hmmm maybe for why the BIDS validator web version and command line version don't agree, the environment has to be updated... If @sappelhoff knows where the install of bids-validator is coming from that gets called by the command line in test_write that would solve that issue otherwise I'll look into.

As far as the issue of the web version bids-validator giving an error for microvolts, that seems to be an error on the bids-validator side as far as I know. I'll open an issue there.

alexrockhill commented 4 years ago

Also maybe @sappelhoff or someone else knows why the bids-validator is called from a subprocess and not using the python API

sappelhoff commented 4 years ago

The BIDS-validator Python API is very limited ... actually, there is not really any. There is only this .py file, which gives a user the functionality to check if a "file" is bids conformant.

re: validator version, we are making a development install of the validator by cloning the repo and checking out at a specific hash (and then installing from that hash using yarn, but see the instructions I linked). Currently that hash is:

VALIDATOR_VERSION: "8887074e5f5f48a768842ffcf96fb2952dbe0e38"

see appveyor and travis

you could make a local development install of bids validator on your machine and then run your test with different versions to find out:

adam2392 commented 4 years ago

@alexrockhill I also faced this issue and didn't go further to figure out what the issue is... so thank you for fixing this :)

alexrockhill commented 4 years ago

Update: using microV passes the validator. Hardly optimal but it is a solution.

jasmainak commented 4 years ago

@alexrockhill microV is not SI abbreviation as far as I remember

alexrockhill commented 4 years ago

The way it's coded in the specification is that both the "name" and the actual SI unit are accepted. Maybe that's a larger problem that should be addressed so that it's just the unit that's accepted.

In bids-validator/bids-validator/utils/unit.js line 62:

const prefixes = [
  // multiples
  'deca',
  'da',
  'hecto',
  'h',
  'kilo',
  'k',
  'mega',
  'M',
  'giga',
  'G',
  'tera',
  'T',
  'peta',
  'P',
  'exa',
  'E',
  'zetta',
  'Z',
  'yotta',
  'Y',
  // sub-multiples
  'deci',
  'd',
  'centi',
  'c',
  'milli',
  'm',
  'micro',
  '\u03BC', // µ
  'nano',
  'n',
  'pico',
  'p',
  'femto',
  'f',
  'atto',
  'a',
  'zepto',
  'z',
  'yocto',
  'y',
]

All these are fed into the regular expression (line 125)

const start = '^'
const end = '$'
const prefix = `(${prefixes.join('|')})?`
const root = `(${roots.join('|')})`
const superscriptExp = `(${superscriptNegative}?[${superscriptNumbers}]+)?`
const operatorExp = `(\\^-?[0-9]+)?`
const unitWithExponentPattern = new RegExp(
  `${start}${prefix}${root}(${superscriptExp}|${operatorExp})${end}`,
)
jasmainak commented 4 years ago

I don't know, feel free to raise an issue on bids validator. But you never say kilog, you always say kg. Using microV just seems a dangerous hack ...

alexrockhill commented 4 years ago

I agree, I wrote a PR for the bids validator that should fix it so that it accepts microvolts (and ohms and degrees which were also not standard characters and presumably would have crashed the validator). I think that's the best solution.

sappelhoff commented 4 years ago

I wrote a PR for the bids validator that should fix it so that it accepts microvolts

thanks @alexrockhill

jasmainak commented 4 years ago

@alexrockhill I'm closing this issue as it seems more like a validator problem. We never faced this with the command-line version of the validator. Please reopen if you disagree and think something needs to be fixed in mne-bids