Configuration TypeError

[ktavabi]

Custom config block contains

...
process_er = True
noise_cov = 'emptyroom'

python ~/Github/mne-study-template/run.py --config ~/.../custom_config.py throws TypeError as such

Traceback (most recent call last): File "/Users/ktavabi/Github/mne-study-template/run.py", line 188, in fire.Fire(process) File "/Users/ktavabi/miniconda3/lib/python3.8/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/Users/ktavabi/miniconda3/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/Users/ktavabi/miniconda3/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/Users/ktavabi/Github/mne-study-template/run.py", line 183, in process _run_script(script_path, config, root_dir, subject, session, task, run) File "/Users/ktavabi/Github/mne-study-template/run.py", line 92, in _run_script runpy.run_path(script_path, run_name='main') File "/Users/ktavabi/miniconda3/lib/python3.8/runpy.py", line 265, in run_path return _run_module_code(code, init_globals, run_name, File "/Users/ktavabi/miniconda3/lib/python3.8/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Users/ktavabi/miniconda3/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/ktavabi/Github/mne-study-template/scripts/init/00-init_derivatives_dir.py", line 10, in import config File "/Users/ktavabi/Github/mne-study-template/config.py", line 1202, in if isinstance(noise_cov, str) and noise_cov != 'emptyroom': TypeError: isinstance() arg 2 must be a type or tuple of types

Suggestion/advice appreciated.

[agramfort]

hi @ktavabi

can you share your custom_config.py file?

I am wondering if you did not call a variable str which overwrites the type string

[ktavabi]

Here you are good sirs:

> if you did not call a variable str which overwrites the type string

the error reads to me as though two args are expected

Ps. a thought while we're at it...

- maybe a mock `config.py` for `sample`may be shipped out with the codebase

EDIT: quotation marks made no difference...

study_name = "paros-bids"
bids_root = "/Users/ktavabi/MEG/paros/bids"
deriv_root = "derivatives/paros"
subjects_dir = "/Users/ktavabi/freesurfer"
interactive = False
crop = [300, 800]
sessions = "all"
task = str = "lexicaldecision"
eeg_bipolar_channels = {
    "HEOG": ("HEOG_left", "HEOG_right"),
    "VEOG": ("VEOG_lower", "VEOG_upper"),
}
ch_types = ['meg']
###############################################################################
# MAXWELL FILTER PARAMETERS
# -------------------------
# done in 01-import_and_maxfilter.py
use_maxwell_filter = True
mf_reference_run = '01'
find_flat_channels_meg =  True
find_noisy_channels_meg =  True
mf_st_duration =  10.0
mf_head_origin = "auto"
mf_reference_run =  None
mf_cal_fname =  None
mf_ctc_fname =  None
###############################################################################
# STIMULATION ARTIFACT
# --------------------
# used in 01-import_and_maxfilter.py
fix_stim_artifact =  False
stim_artifact_tmin =  0.0
stim_artifact_tmax =  0.01
###############################################################################
# FREQUENCY FILTERING
# -------------------
# done in 02-frequency_filter.py
l_freq =  None
h_freq =  55.0
###############################################################################
# RESAMPLING
# ----------
resample_sfreq = 500
decim = 1
###############################################################################
# AUTOMATIC REJECTION OF ARTIFACTS
# --------------------------------
reject = {"grad": 4000e-13, "mag": 4e-12, "eeg": 150e-6}
reject_tmin = -0.2
reject_tmax = 1.3
###############################################################################
# RENAME EXPERIMENTAL EVENTS
# --------------------------
rename_events = dict()
###############################################################################
# EPOCHING
# --------
conditions = ["lexical", "nonlexical"]
conditions = ["lexical/high", "lexical/low"]
epochs_tmin = -0.2
epochs_tmax = 1.3
baseline = (None, 0)
contrasts = [("lexical", "nonlex"), ("lexical/high", "lexical/low")]
###############################################################################
# ARTIFACT REMOVAL
# ----------------
use_ssp = True
###############################################################################
# DECODING
# --------
decode = True
decoding_metric = "roc_auc"
decoding_n_splits = 5
n_boot = 5000
###############################################################################
# GROUP AVERAGE SENSORS
# ---------------------
interpolate_bads_grand_average = True
###############################################################################
# TIME-FREQUENCY
# --------------
time_frequency_conditions = ["lexical", "nonlex"]
###############################################################################
# SOURCE ESTIMATION PARAMETERS
# ----------------------------
bem_mri_images = "auto"
recreate_bem = True
spacing = "oct6"
mindist = float = 5
inverse_method = "dSPM"
process_er = True
noise_cov = "emptyroom"
###############################################################################
# ADVANCED
# --------
l_trans_bandwidth = "auto"
h_trans_bandwidth = "auto"
N_JOBS = 12
shortest_event = int = 1
allow_maxshield = bool = False
log_level = "info"
mne_log_level = "error"
on_error = "debug"

[agramfort]

thx for sharing

the MNE sample dataset is ds000248 on openneuro. See the config used in tests:

https://github.com/mne-tools/mne-study-template/blob/master/tests/configs/config_ds000248.py https://github.com/mne-tools/mne-study-template/blob/master/tests/configs/config_ds000248_FLASH_BEM.py https://github.com/mne-tools/mne-study-template/blob/master/tests/configs/config_ds000248_T1_BEM.py

HTH

[ktavabi]

Out of the box: 4ec9276

python tests/run_tests.py --download 0 ds000248
Running the following tests: ds000248

throws ValueError...

13:01:35 Using custom configuration: /Users/ktavabi/Github/mne-study-template/tests/configs/config_ds000248.py Traceback (most recent call last): File "tests/run_tests.py", line 120, in run_tests(test_suite, download=download) File "tests/run_tests.py", line 87, in run_tests runpy.run_path(run_script, run_name='main') File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/runpy.py", line 265, in run_path return _run_module_code(code, init_globals, run_name, File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/ktavabi/Github/mne-study-template/run.py", line 188, in fire.Fire(process) File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/Users/ktavabi/Github/mne-study-template/run.py", line 183, in process _run_script(script_path, config, root_dir, subject, session, task, run) File "/Users/ktavabi/Github/mne-study-template/run.py", line 92, in _run_script runpy.run_path(script_path, run_name='main') File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/runpy.py", line 265, in run_path return _run_module_code(code, init_globals, run_name, File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/ktavabi/Github/mne-study-template/scripts/init/00-init_derivatives_dir.py", line 10, in import config File "/Users/ktavabi/Github/mne-study-template/config.py", line 1270, in raise ValueError(msg) ValueError: You set mf_reference_run=01, but your dataset only contains the following runs: [None]

[agramfort]

it's weird as the CI is green.

what does mne.sys_info() tell you?

[ktavabi]

Platform: macOS-10.16-x86_64-i386-64bit Python: 3.8.8 (default, Feb 24 2021, 13:46:16) [Clang 10.0.0 ] Executable: /Users/ktavabi/miniconda3/envs/obiwan/bin/python CPU: i386: 12 cores Memory: 32.0 GB mne: 0.22.dev0 numpy: 1.20.0 {blas=openblas, lapack=openblas} scipy: 1.5.3 matplotlib: 3.3.4 {backend=MacOSX} sklearn: 0.24.1 numba: 0.53.0 nibabel: 3.2.1 cupy: Not found pandas: 1.2.3 dipy: 1.3.0 mayavi: Not found pyvista: 0.27.3 {pyvistaqt=0.2.0, OpenGL 4.1 ATI-4.2.15 via AMD Radeon Pro Vega 16 OpenGL Engine} vtk: 9.0.1 PyQt5: 5.12.3

PS my VM IDE (VSCode) is throwing this weirdo:

SyntaxError: Non-ASCII character '\xe2' in file /Users/ktavabi/Github/mne-study-template/run.py on line 62, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

[hoechenberger]

PS my VM IDE (VSCode) is throwing this weirdo:

SyntaxError: Non-ASCII character '\xe2' in file /Users/ktavabi/Github/mne-study-template/run.py on line 62, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

This is the Unicode character "–" (en-dash), used in a comment.

It's interesting Python complains for you, as Unicode is the default for Python 3 files … Do you have any of the LOCALE, LC_CTYPE, LC_ALL, or LANGUAGE environment variables set?

[ktavabi]

None of those vars are currently set; nor did I know of them to set them explicitly...

It's interesting Python complains for you

@hoechenberger I'm as stupid as they come, I like my environments like I like my iRobot roombas

[agramfort]

it's really weird.

do you maybe have 2 version of python available on your system?

when you do

python --version

in the same terminal that crashes what do you get?

[hoechenberger]

python tests/run_tests.py --download 0 ds000248

Could you try removing the dataset (rm -rf ~/mne_data/ds000248) and then running the test again via

python tests/run_tests.py ds000248

to force re-downloading?

[hoechenberger]

PS my VM IDE (VSCode) is throwing this weirdo:

SyntaxError: Non-ASCII character '\xe2' in file /Users/ktavabi/Github/mne-study-template/run.py on line 62, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

I'd almost assume that VS Code, here, is simply using the wrong Python environment – some global Python 2.7 interpreter.

[ktavabi]

Before and after error I'm in a 'conda' venv with Python 3.8.8 @agramfort

@hoechenberger python tests/run_tests.py ds000248

ModuleNotFoundError: No module named 'openneuro'

pip install openneuro-py and error went from bad to worst...

A critical error occurred. Traceback (most recent call last): File "tests/run_tests.py", line 120, in run_tests(test_suite, download=download) File "tests/run_tests.py", line 87, in run_tests runpy.run_path(run_script, run_name='main') File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/runpy.py", line 265, in run_path return _run_module_code(code, init_globals, run_name, File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/ktavabi/Github/mne-study-template/run.py", line 188, in fire.Fire(process) File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, *kwargs) File "/Users/ktavabi/Github/mne-study-template/run.py", line 183, in process _run_script(script_path, config, root_dir, subject, session, task, run) File "/Users/ktavabi/Github/mne-study-template/run.py", line 92, in _run_script runpy.run_path(script_path, run_name='main') File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/runpy.py", line 265, in run_path return _run_module_code(code, init_globals, run_name, File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/ktavabi/Github/mne-study-template/scripts/preprocessing/01-import_and_maxfilter.py", line 462, in main() File "/Users/ktavabi/Github/mne-study-template/config.py", line 1402, in wrapper raise(e) File "/Users/ktavabi/Github/mne-study-template/config.py", line 1395, in wrapper return func(args, **kwargs) File "/Users/ktavabi/Github/mne-study-template/scripts/preprocessing/01-import_and_maxfilter.py", line 454, in main parallel(run_func(subject, session) for subject, session in File "/Users/ktavabi/Github/mne-study-template/scripts/preprocessing/01-import_and_maxfilter.py", line 454, in parallel(run_func(subject, session) for subject, session in File "/Users/ktavabi/Github/mne-study-template/scripts/preprocessing/01-import_and_maxfilter.py", line 376, in run_maxwell_filter raw_out.save(raw_fname_out, picks=chs_to_include, overwrite=True, File "", line 21, in save File "/Users/ktavabi/Github/mne-python/mne/io/base.py", line 1405, in save _write_raw(fname, self, info, picks, fmt, data_type, reset_range, File "/Users/ktavabi/Github/mne-python/mne/io/base.py", line 1864, in _write_raw use_fname = _construct_bids_filename(base, ext, part_idx) File "/Users/ktavabi/Github/mne-python/mne/io/utils.py", line 309, in _construct_bids_filename use_fname = '{}split-{:02}{}{}'.format(base, part_idx, modality, ext) UnboundLocalError: local variable 'modality' referenced before assignment

[agramfort]

there must be something messed up with your system python and your environments.

did you install openneuro-py in the current env?

[ktavabi]

did you install openneuro-py in the current env?

yup. openneuro-py 2021.2on

conda environments:
base                     /Users/ktavabi/miniconda3
obiwan                *  /Users/ktavabi/miniconda3/envs/obiwan
tatooine                 /Users/ktavabi/miniconda3/envs/tatooine

there must be something messed up with your system python and your environments.

Could you be more specific Master Skywalker?

[agramfort]

it's impossible that you get

ModuleNotFoundError: No module named 'openneuro'

if openneuro-py is installed.

the env you are working in must have some issue with missing dependencies

Alex

[ktavabi]

 3/24/2021 12:41  conda create -n foobar python=3.8 anaconda
 3/24/2021 12:44  conda activate foobar
 3/24/2021 12:45  pip install mne-bids coloredlogs tqdm pandas scikit-learn json_tricks fire
 3/24/2021 12:47  cd ~/Github/mne-study-template
 3/24/2021 12:47  rm -rf ~/mne_data/ds000248
 3/24/2021 12:48  python tests/run_tests.py ds000248
 3/24/2021 12:55  pip install mne
 3/24/2021 12:56  python tests/run_tests.py ds000248
.
.
.
Running the following tests: ds000248
Traceback (most recent call last):
  File "tests/run_tests.py", line 120, in <module>
    run_tests(test_suite, download=download)
  File "tests/run_tests.py", line 73, in run_tests
    fetch(dataset)
  File "tests/run_tests.py", line 15, in fetch
    from download_test_data import main
  File "/Users/ktavabi/Github/mne-study-template/tests/download_test_data.py", line 4, in <module>
    import openneuro
ModuleNotFoundError: No module named 'openneuro'

FWIW openneuro is not even listed as a dependency

[agramfort]

can you do also

pip install openneuro-py

?

[hoechenberger]

Hello, openneuro-py is not listed as a dependency because it's only a "dev dependency". But this should probably be documented, yes.

[ktavabi]

New build as of today using Python 3.9.2 now and everything from https://raw.githubusercontent.com/mne-tools/mne-python/master/environment.yml including pip flavor of oppenneuro-py...

16:14:42 Using custom configuration: /Users/ktavabi/Github/mne-study-template/tests/configs/config_ds000248.py 16:14:42 [Step-01] Running: Initializing output directories. 16:14:42 [Step-01] Initializing output directories. 16:14:42 [Step-01] Completed: Initializing output directories. 2021-03-24 16:14:42 INFO Successfully finished running: init_derivatives_dir 2021-03-24 16:14:42 INFO Now running: import_and_maxfilter refvol.fsl_rigid_register.nii: 45%|█▊ | 7.28M/16.0M [00:14<00:00, 12.0MB/s]16:14:54 [Step-01] Running Step 1: Data import and Maxwell filtering 16:14:54 [Step-01][sub-01] Loading reference run: 01. 16:14:54 A critical error occurred. Traceback (most recent call last): File "/Users/ktavabi/Github/mne-study-template/tests/run_tests.py", line 120, in run_tests(test_suite, download=download) File "/Users/ktavabi/Github/mne-study-template/tests/run_tests.py", line 87, in run_tests runpy.run_path(run_script, run_name='main') File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.9/runpy.py", line 268, in run_path return _run_module_code(code, init_globals, run_name, File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.9/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/ktavabi/Github/mne-study-template/run.py", line 188, in fire.Fire(process) File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.9/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.9/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.9/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, *kwargs) File "/Users/ktavabi/Github/mne-study-template/run.py", line 183, in process _run_script(script_path, config, root_dir, subject, session, task, run) File "/Users/ktavabi/Github/mne-study-template/run.py", line 92, in _run_script runpy.run_path(script_path, run_name='main') File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.9/runpy.py", line 268, in run_path return _run_module_code(code, init_globals, run_name, File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.9/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/ktavabi/Github/mne-study-template/scripts/preprocessing/01-import_and_maxfilter.py", line 462, in main() File "/Users/ktavabi/Github/mne-study-template/config.py", line 1402, in wrapper raise(e) File "/Users/ktavabi/Github/mne-study-template/config.py", line 1395, in wrapper return func(args, **kwargs) File "/Users/ktavabi/Github/mne-study-template/scripts/preprocessing/01-import_and_maxfilter.py", line 454, in main parallel(run_func(subject, session) for subject, session in File "/Users/ktavabi/Github/mne-study-template/scripts/preprocessing/01-import_and_maxfilter.py", line 454, in parallel(run_func(subject, session) for subject, session in File "/Users/ktavabi/Github/mne-study-template/scripts/preprocessing/01-import_and_maxfilter.py", line 301, in run_maxwell_filter info = mne.io.read_info(bids_path_in.fpath) File "", line 24, in read_info File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.9/site-packages/mne/io/meas_info.py", line 917, in readinfo f, tree, = fiff_open(fname) File "", line 24, in fiff_open File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.9/site-packages/mne/io/open.py", line 125, in fiff_open fid = _fiff_get_fid(fname) File "/Users/ktavabi/miniconda3/envs/obiwan/lib/python3.9/site-packages/mne/io/open.py", line 54, in _fiff_get_fid fid = open(fname, "rb") # Open in binary mode FileNotFoundError: [Errno 2] No such file or directory: '/Users/ktavabi/mne_data/ds000248/sub-01/meg/sub-01_task-audiovisual_run-01_meg'

[agramfort]

have you downloaded the files? where is the dataset ds000248 on your disk?

[ktavabi]

😐

ls ~/mne_data/ds000248
CHANGES                  dataset_description.json participants.json        sub-emptyroom
README                   derivatives              sub-01

Forget it AG. If you say it works, then it must.