search

int-brain-lab / ONE

Open Neurophysiology Environment
MIT License
17 stars 5 forks source link

load_datasets fails when passed paths containing revisions #123

Closed k1o0 closed 2 weeks ago

k1o0 commented 5 months ago

Related to #65 

from one.api import ONE
one_cache = ONE()
eid = '004d8fd5-41e7-4f1b-a45b-0d4ad76fe446'
one = ONE(base_url='https://alyx.internationalbrainlab.org/', mode='remote', cache_dir=one_cache.cache_dir.parent / 'foo')
dsets = one.list_datasets(eid, collection='alf/probe00/pykilosort', revision='2024-05-06')
files = one.load_datasets(eid, dsets, download_only=True)

This chains a call to list_datasets that in turn filters out the revision as it is set to None, which causes an object not found exception.

oliche commented 4 months ago

Analysis and todos from today w/ Miles:

from one.api import ONE
from one.util import filter_datasets
one_cache = ONE()
eid = '004d8fd5-41e7-4f1b-a45b-0d4ad76fe446'
one = ONE(base_url='https://alyx.internationalbrainlab.org/', mode='remote', cache_dir=one_cache.cache_dir.parent / 'foo')

dsets = one.list_datasets(eid, collection='alf/probe00/pykilosort', revision='2024-05-06', filename='channels*')
files = one.load_datasets(eid, dsets, download_only=True, revisions='2024-05-06')  # this should fail if there is no dataframe
files = one.load_datasets(eid, dsets, download_only=True)  # this should work
files = one.load_datasets(eid, dsets, download_only=True, revisions='2024-10-10')  # this should fail as no dataset matches now

df_dsets = one.list_datasets(eid, collection='alf/probe00/pykilosort', revision='2024-05-06', filename='channels*', details=True)
assert df_dsets.revision.unique() == '2024-05-06'
files = one.load_datasets(eid, df_dsets, download_only=True)  # this should work
# todo raise if any of revisions / collections are provided with a dataframe (any filter)
# todo deactivate the filter datasets if a dataframe OR a relative path is provided
# todo raise if revision singular is provided as kwarg

dsets = one.list_datasets(eid, collection='alf/probe00/pykilosort', filename='channels*')   # TODO raise warning with mixed revisions  / mixed defaults (check existing warnings)
dsets = one.list_datasets(eid, collection='alf/probe00/pykilosort', revision='*', filename='channels*')  # doing the last before and then restricts to this revision

# mock checkfilesystems for tests mock.patch.object(one, 'download_datasets')
# check that behaviour is consistent with load_dataset
# todo load_dataset should raise if kwarg filters are provided with a dataframe record