[Bug report] - phylib cannot access the raw ephys file

[LiuDaveLiu]

Describe the bug PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'H:\dl62\raw_ephys_data\_spikeglx_ephysData_g0_t0.imec.ap.bin'

I used iblapps.atlaselectrophysiology.extract_files.extract_data to extract data offline, and ran into this error on valid WindowsPath.

To Reproduce Steps to reproduce the behavior:

1. Open Anaconda prompt
2. activate iblenv
3. from pathlib import Path
4. from iblapps.atlaselectrophysiology.extract_files import extract_data
5. ks_path = Path('H:\alfExtract\dl62\20190131_1\KS2'); ephys_path = Path('H:\alfExtract\dl62\20190131_1\raw_ephys_data'); out_path = Path('H:\alfExtract\dl62\20190131_1\alf')

6. Error below

   PermissionError Traceback (most recent call last)

   in ----> 1 extract_data(ks_path, ephys_path, out_path)

~\int-brain-lab\iblapps\atlaselectrophysiology\extract_files.py in extract_data(ks_path, ephys_path, out_path) 109 if efile.get('ap') and efile.ap.exists(): 110 ks2_to_alf(ks_path, ephys_path, out_path, bin_file=efile.ap, --> 111 ampfactor=_sample2v(efile.ap), label=None, force=True) 112 extract_rmsmap(efile.ap, out_folder=out_path, spectra=False) 113 if efile.get('lf') and efile.lf.exists():

~\int-brain-lab\ibllib-repo\ibllib\ephys\spikes.py in ks2_to_alf(ks_path, bin_path, out_path, bin_file, ampfactor, label, force) 174 :return: 175 """ --> 176 m = ephysqc.phy_model_from_ks2_path(ks2_path=ks_path, bin_path=bin_path, bin_file=bin_file) 177 ephysqc.unit_metrics_ks2(ks_path, m, save=True) 178 ac = alf.EphysAlfCreator(m)

~\int-brain-lab\ibllib-repo\ibllib\ephys\ephysqc.py in phy_model_from_ks2_path(ks2_path, bin_path, bin_file) 275 sample_rate=fs, 276 n_channels_dat=nch, --> 277 n_closest_channels=NCH_WAVEFORMS) 278 m.depths = m.get_depths() 279 return m

~\int-brain-lab\phylib\phylib\io\model.py in init(self, **kwargs) 259 self.offset = getattr(self, 'offset', 0) 260 --> 261 self._load_data() 262 263 #--------------------------------------------------------------------------

~\int-brain-lab\phylib\phylib\io\model.py in _load_data(self) 357 358 # Traces and duration. --> 359 self.traces = self._load_traces(self.channel_mapping) 360 if self.traces is not None: 361 self.duration = self.traces.duration

~\int-brain-lab\phylib\phylib\io\model.py in _load_traces(self, channel_map) 490 traces = get_ephys_reader( 491 self.dat_path, n_channels_dat=n, dtype=self.dtype, offset=self.offset, --> 492 sample_rate=self.sample_rate) 493 if traces is not None: 494 traces = traces[:, channel_map] # lazy permutation on the channel axis

~\int-brain-lab\phylib\phylib\io\traces.py in get_ephys_reader(obj, kwargs) 492 if not klass: 493 return --> 494 return klass(arg, kwargs) 495 496

~\int-brain-lab\phylib\phylib\io\traces.py in init(self, paths, sample_rate, dtype, offset, n_channels, **kwargs) 314 self._mmaps = [ 315 _memmap_flat(path, dtype=dtype, n_channels=n_channels, offset=offset) --> 316 for path in paths] 317 318 self.sample_rate = sample_rate

~\int-brain-lab\phylib\phylib\io\traces.py in (.0) 314 self._mmaps = [ 315 _memmap_flat(path, dtype=dtype, n_channels=n_channels, offset=offset) --> 316 for path in paths] 317 318 self.sample_rate = sample_rate

~\int-brain-lab\phylib\phylib\io\traces.py in _memmap_flat(path, dtype, n_channels, offset) 161 n_samples = (fsize - offset) // (item_size * n_channels) 162 shape = (n_samples, n_channels) --> 163 return np.memmap(path, dtype=dtype, offset=offset, shape=shape) 164 165

C:\ProgramData\Anaconda3\envs\iblenv\lib\site-packages\numpy\core\memmap.py in new(subtype, filename, dtype, mode, offset, shape, order) 273 # special case - if we were constructed with a pathlib.path, 274 # then filename is a path object, not a string --> 275 self.filename = filename.resolve() 276 elif hasattr(fid, "name") and isinstance(fid.name, str): 277 # py3 returns int for TemporaryFile().name

C:\ProgramData\Anaconda3\envs\iblenv\lib\pathlib.py in resolve(self, strict) 1159 if self._closed: 1160 self._raise_closed() -> 1161 s = self._flavour.resolve(self, strict=strict) 1162 if s is None: 1163 # No symlink resolution => for consistency, raise an error if

C:\ProgramData\Anaconda3\envs\iblenv\lib\pathlib.py in resolve(self, path, strict) 203 while True: 204 try: --> 205 s = self._ext_to_normal(_getfinalpathname(s)) 206 except FileNotFoundError: 207 previous_s = s

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'H:\dl62\raw_ephys_data\_spikeglx_ephysData_g0_t0.imec.ap.bin'

Expected behavior Error when attempting to access the file.

Screenshots

Desktop (please complete the following information):

• OS: Windows 7
• Python 3.7.4

Additional context Running the same data extraction steps on Mac OS with Python 3.8.3 did not result in any errors.

[rossant]

• I wonder if you shouldn't try to add a r just between ( and ' here (otherwise backslash might be incorrectly interpreted as an escape symbol, leading to a wrong path, but Windows might give you a confusing error message):

  ks_path = Path(r'H:\alfExtract\dl62\20190131_1\KS2');
  ephys_path = Path(r'H:\alfExtract\dl62\20190131_1\raw_ephys_data');
  out_path = Path(r'H:\alfExtract\dl62\20190131_1\alf')

• If that doesn't work, are you able to rename the file in Windows?

• Can you move/copy it to another drive? If so, does the error persist when you update the path to the file accordingly?

[LiuDaveLiu]

Thanks for the suggestions,

• I tried adding a r between ( and ' and that didn't work. ephys_path.resolve() gives the correct WindowsPath.
• I was not able to rename the file in Windows, and Windows gave me the warning that the file is open in Python.
• After closing Python, I can move it to another drive, and the error persists when I update the path to the file accordingly.

[rossant]

Could you try the following steps:

1. Restart your computer
2. Launch a Jupyter notebook
3. Type the following:

import os
import stat
import numpy as np

# new cell
path = r'H:\dl62\raw_ephys_data\_spikeglx_ephysData_g0_t0.imec.ap.bin'
st = os.stat(path)
print(st.st_mode)

# new cell
X = np.memmap(path, dtype=np.int16)
X

4. Post the output

[LiuDaveLiu]

Here is the output, By the way, I pulled the latest repos and the issue persists.