search

flatironinstitute / spikeforest_old

SpikeForest -- spike sorting analysis for website
Apache License 2.0
16 stars 8 forks source link

Download problems #37

Open samuelgarcia opened 5 years ago

samuelgarcia commented 5 years ago

Hi all, I try to download locally with:

recdir = 'sha1dir://fb52d510d2543634e247e0d2d1d4390be9ed9e20.synth_magland/datasets_noise10_K10_C4/001_synth'
recording = SFMdaRecordingExtractor(dataset_directory=recdir, download=True)
sorting_gt = SFMdaSortingExtractor(firings_file=recdir + '/firings_true.mda')

but I have this 

Exception: Unable to find file: sha1dir://fb52d510d2543634e247e0d2d1d4390be9ed9e20.synth_magland/datasets_noise10_K10_C4/001_synth/params.json

Does the sha1 key sometimes ?

samuelgarcia commented 5 years ago

OUps I was missing this: mt.configDownloadFrom('spikeforest.public')

samuelgarcia commented 5 years ago

sorry

samuelgarcia commented 5 years ago

I finally have some other donwload problems. I am trying to make a loop to donwload a dataset.

recdir = 'sha1dir://fb52d510d2543634e247e0d2d1d4390be9ed9e20.synth_magland/datasets_noise10_K10_C4/001_synth'

is OK

But 

recdir = 'sha1dir://fb52d510d2543634e247e0d2d1d4390be9ed9e20.synth_magland/datasets_noise10_K10_C4/002_synth'

fails.

Any ideas ?

Error:

sha1dir://fb52d510d2543634e247e0d2d1d4390be9ed9e20.synth_maglang/datasets_noise10_K10_C4/002_synth
Traceback (most recent call last):
  File "download_from_SF.py", line 78, in <module>
    download_one_study(familly, study)
  File "download_from_SF.py", line 46, in download_one_study
    recording = SFMdaRecordingExtractor(dataset_directory=recdir, download=True)
  File "/home/samuel/.virtualenvs/py36/lib/python3.6/site-packages/spikeforest/extractors/sfmdaextractors/sfmdaextractors.py", line 38, in __init__
    raise Exception('Unable to realize file: ' + self._timeseries_path)
Exception: Unable to realize file: sha1dir://fb52d510d2543634e247e0d2d1d4390be9ed9e20.synth_maglang/datasets_noise10_K10_C4/002_synth/raw.mda
magland commented 5 years ago

Only some recordings are publicly available. See https://github.com/flatironinstitute/spikeforest/blob/master/docs/tutorials/spike_sorting_spikeforest_recording.md

samuelgarcia commented 5 years ago

OK.I see, I should read more that the first 5 lines of example before coding :) I though it was totally open. What is the reason of not opening the entire datset ?

For instance, for some dataset having one unique file is useless because there is only one GT. For instance, PAIRED_MEA64C_YGER/paired_mea64c/20160415_patch2 In that case, we need several files to make stats.. Nevertheless they can be download directly at zenodo.

magland commented 5 years ago

Thanks Samuel. Yes if they are available on zenodo they are public anyway. As of now (recently changed), all of the paired_* recordings are public, with the exception of boyden recordings as I'm not sure of the status of the conditions for those recordings.

Regarding the other recordings (e.g., simulated datasets)... we are still thinking about public (training) vs holdout (validation) recordings. This is something that needs to be addressed more carefully.