MKL FFT error from process_nwb.resample

jihyunbak commented 3 years ago

This is what I get while testing PR #85 , so you need to be on that branch to reproduce this.

Specifically, running test_build_nwb.py for blocks RVG16_B06 (Tone) or RVG16_B07 (Tone150), with resample_data=True:

(nsds_nwb) [jhbak@catscan tests]$ python -u test_build_nwb.py 
INFO:nsds_lab_to_nwb.nwb_builder:Collecting metadata for NWB conversion...
INFO:nsds_lab_to_nwb.nwb_builder:Collecting relevant input data paths...
INFO:nsds_lab_to_nwb.nwb_builder:Preparing output path...
INFO:nsds_lab_to_nwb.nwb_builder:Creating originator instances...
/clusterfs/bebb/users/jhbak/opt/miniconda3/envs/nsds_nwb/lib/python3.7/site-packages/tdt/TDTbin2py.py:884: ResourceWarning: unclosed file <_io.TextIOWrapper name='/clusterfs/NSDS_data/raw/RVG16/RVG16_B06/TDT_Tanks_RVG16_B6.tnt' mode='r' encoding='UTF-8'>
  lines = np.array([line.rstrip('\n') for line in open(tnt_path)])
ResourceWarning: Enable tracemalloc to get the object allocation traceback
read from t=0s to t=1463.58s
INFO:nsds_lab_to_nwb.nwb_builder:Extracting session start time...
INFO:nsds_lab_to_nwb.nwb_builder:Building components for NWB
INFO:nsds_lab_to_nwb.nwb_builder:Adding electrode information...
INFO:nsds_lab_to_nwb.nwb_builder:Adding neural data...
E
======================================================================
ERROR: test_build_nwb_single_block (__main__.TestCase_Build_NWB)
build NWB but do not write file to disk
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_build_nwb.py", line 34, in test_build_nwb_single_block
    self.__build_nwb_content(block_folder, resample_data, use_htk)
  File "test_build_nwb.py", line 48, in __build_nwb_content
    nwb_content = nwb_builder.build()
  File "/pool0/clusterfs/bebb/users/jhbak/proj/nsds_lab_to_nwb/nsds_lab_to_nwb/nwb_builder.py", line 173, in build
    self.neural_data_originator.make(nwb_content, electrode_table_regions)
  File "/pool0/clusterfs/bebb/users/jhbak/proj/nsds_lab_to_nwb/nsds_lab_to_nwb/components/neural_data/neural_data_originator.py", line 77, in make
    data = self.resample(data)
  File "/pool0/clusterfs/bebb/users/jhbak/proj/nsds_lab_to_nwb/nsds_lab_to_nwb/components/neural_data/neural_data_originator.py", line 50, in resample
    new_data = resample(data, new_freq, rate)
  File "/clusterfs/bebb/users/jhbak/opt/miniconda3/envs/nsds_nwb/lib/python3.7/site-packages/process_nwb/resample.py", line 122, in resample
    Xds = resample_func(X, new_n_time, npad=npad, real=real)
  File "/clusterfs/bebb/users/jhbak/opt/miniconda3/envs/nsds_nwb/lib/python3.7/site-packages/process_nwb/resample.py", line 77, in resample_func
    X_fft = rfft(X, axis=0)
  File "/clusterfs/bebb/users/jhbak/opt/miniconda3/envs/nsds_nwb/lib/python3.7/site-packages/mkl_fft/_numpy_fft.py", line 414, in rfft
    (x,), {'n': n, 'axis': axis})
  File "/clusterfs/bebb/users/jhbak/opt/miniconda3/envs/nsds_nwb/lib/python3.7/site-packages/mkl_fft/_numpy_fft.py", line 103, in trycall
    raise ve
  File "/clusterfs/bebb/users/jhbak/opt/miniconda3/envs/nsds_nwb/lib/python3.7/site-packages/mkl_fft/_numpy_fft.py", line 98, in trycall
    res = func(*args, **kwrds)
  File "mkl_fft/_pydfti.pyx", line 792, in mkl_fft._pydfti.rfft_numpy
  File "mkl_fft/_pydfti.pyx", line 701, in mkl_fft._pydfti._rc_fft1d_impl
ValueError: Internal error occurred: b'Intel MKL DFTI ERROR: Inconsistent configuration parameters'

----------------------------------------------------------------------
Ran 1 test in 143.069s

FAILED (errors=1)

JesseLivezey commented 3 years ago

@jihyunbak can you try installing process_nwb from github source? I think I fixed this, but maybe need to create a new pypi version for pip.

jihyunbak commented 3 years ago

I tried pip install git+https://github.com/BouchardLab/process_nwb.git but same error …

Next I cloned the process_nwb repo and did pip install -e . in the repo, but somehow test is taking too long; let me update once the test runs.

JesseLivezey commented 3 years ago

@jihyunbak resampling is going to be the slowest step (by far) for long blocks, especially for ones that hit that MKL error.

jihyunbak commented 3 years ago

Test ran successfully after I installed in the editable mode from the locally cloned process_nwb repo!

BouchardLab / nsds_lab_to_nwb

MKL FFT error from process_nwb.resample #86