int-brain-lab / ONE

Open Neurophysiology Environment
MIT License
16 stars 4 forks source link

alfio Find variants reports non ambiguous datasets as ambiguous and raises #126

Closed oliche closed 1 day ago

oliche commented 2 weeks ago

For an input signature of

[('*ap.meta', 'raw_ephys_data/probe01', True),
 ('*ap.cbin', 'raw_ephys_data/probe01', True),
 ('*ap.ch', 'raw_ephys_data/probe01', True),
 ('*ap.wiring.json', 'raw_ephys_data/probe01', False),
 ('_spikeglx_sync.times.*npy', 'raw_ephys_data/probe00', True),
 ('_spikeglx_sync.polarities.*npy', 'raw_ephys_data/probe00', True),
 ('_spikeglx_sync.channels.*npy', 'raw_ephys_data/probe00', True),
 ('*.ap.meta', 'raw_ephys_data/probe00', False)]

And a folder containing:

_spikeglx_ephysData_g0_t0.imec.ap.meta
_spikeglx_ephysData_g0_t0.imec.lf.meta
_spikeglx_sync.channels.probe00.npy
_spikeglx_sync.polarities.probe00.npy
_spikeglx_sync.times.probe00.npy

The extraction gets prevented because ap.meta and lf.meta are reported as ambiguous

2024-09-13 10:13:10 ERROR    tasks.py:459  Ambiguous input datasets found: {'raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec.ap.meta': ['raw_ephys_data/probe00/_spikeglx_ephysData_g0_t0.imec.lf.meta']}
2024-09-13 10:13:10 ERROR    tasks.py:241  Traceback (most recent call last):
  File "/home/olivier/PycharmProjects/pykilosort/ibllib/ibllib/pipes/tasks.py", line 212, in run
    setup = self.setUp(**kwargs)
            ^^^^^^^^^^^^^^^^^^^^
  File "/home/olivier/PycharmProjects/pykilosort/ibllib/ibllib/pipes/tasks.py", line 389, in setUp
    self.assert_expected_inputs()
  File "/home/olivier/PycharmProjects/pykilosort/ibllib/ibllib/pipes/tasks.py", line 463, in assert_expected_inputs
    raise NotImplementedError(
NotImplementedError: Multiple variant datasets found. Loading for these is undefined.
k1o0 commented 1 day ago

Looks like the ONE code is fine. You can resolve this issue in ibllib by removing extra=False from this line. I believe I added that because I was worried about the UUID filenames but given that we symlink only those files that exist in the cache this should not be an issue for running tasks on FlatIron.