64bit firings.mda write error with big files

[shashwatsridhar]

Hey,

I have just begun using mountainlab-js for spike sorting. I followed the installation steps and have mountainlab-js installed along with the following packages installed.

banjoview.cross_correlograms ephys.bandpass_filter ephys.compare_ground_truth ephys.compute_cluster_metrics ephys.compute_cross_correlograms ephys.compute_templates ephys.convert_array ephys.synthesize_random_firings ephys.synthesize_random_waveforms ephys.synthesize_timeseries ephys.whiten kbucket.download kbucket.upload mountainsortalg.ms3 mountainsortalg.ms3alg ms3.apply_timestamp_offset ms3.apply_whitening_matrix ms3.bandpass_filter ms3.cluster_metrics ms3.combine_cluster_metrics ms3.combine_firing_segments ms3.combine_firings ms3.compute_amplitudes ms3.compute_templates ms3.compute_whitening_matrix ms3.concat_event_times ms3.concat_firings ms3.concat_timeseries ms3.confusion_matrix ms3.create_firings ms3.create_multiscale_timeseries ms3.extract_clips ms3.extract_firings ms3.isolation_metrics ms3.link_segments ms3.load_test ms3.mask_out_artifacts ms3.mv_compute_amplitudes ms3.mv_compute_templates ms3.mv_discrimhist ms3.mv_extract_clips ms3.mv_extract_clips_features ms3.mv_subfirings ms3.reorder_labels ms3.run_metrics_script ms3.split_firings ms3.whiten ms3.whiten_clips ms4alg.sort mv.compute_templates mv.create_multiscale_timeseries mv.extract_clips mv.mv_compute_amplitudes mv.mv_compute_templates mv.mv_discrimhist mv.mv_extract_clips mv.mv_extract_clips_features mv.mv_subfirings pyms.apply_label_map pyms.compute_accuracies pyms.compute_templates pyms.create_label_map pyms.extract_clips pyms.extract_geom pyms.extract_timeseries pyms.normalize_channels pyms.synthesize_drifting_timeseries pyms.synthesize_random_firings pyms.synthesize_random_waveforms pyms.synthesize_timeseries spikeview.metrics1 spikeview.templates

As you can see, I have packages from the old implementation of mountainsort linked to the new installation as well. I run the ephys.bandpass_filter followed by ms4alg.sort on my data using the following command.

ml-run-process ephys.bandpass_filter --inputs timeseries:i140703-001.mda --outputs timeseries_out:i140703-001-filt.mda --parameters samplerate:30000 freq_min:300 freq_max:6000 && ml-run-process ms4alg.sort --inputs timeseries:i140703-001-filt.mda geom:utah_geom.csv --outputs firings_out:i140710-001-firings.mda --parameters adjacency_radius:0 detect_sign:-1 detect_threshold:2.5 clip_size:40

The filtering works just fine and creates a filtered data file. I know it works fine, because I use the ev-view-timeseries package to visualize the data. But, the second step seems to work alright all the way to the end, but then doesn't really work. Here's why.

Visualizing it using qt-mountainview simply doesn't work. I use the following command to initiate it:

qt-mountainview --raw=i140703-001.mda --filt=i140703-001-filt.mda --geom=utah_geom.csv --firings=i140703-001-firings.mda

It appears to do a lot of computation, but the command-line output starts so:

** qt-mountainview ; origin: https://github.com/flatironinstitute/qt-mountainview.git ; commit: tags/standalone-r1-0-g9a3acac ** Compiled using Qt version: 5.5.1 (/usr on host: 'inm6058' Setting up object registry... Parsing command-line parameters... Creating MVContext... Setting up context... Creating prv object for: i140703-001-filt.mda Creating prv object for: i140703-001.mda Creating prv object for: i140703-001-firings.mda Setting up main window... Adding controls to main window... Opening initial views... Starting event loop...

and then thousands of lines of:

Warning problem reading chunk in diskreadmda: 0<>100000

and the qt-mountainview window shows nothing. This might be a qt-mountainview error, so I'd be happy to file another report on the qt-mountainview repository.

I then used the readmda function to load the firings file and it throws up this error:

cannot reshape array of size 0 into shape (3,109122729)

so then I used numpy to load the same file using numpy.fromfile() with count=-1 and the output is just:

array([1.90979621e-313, 6.36598737e-314])

So, finally, I checked the filesize of the firings file and its just 20 bytes. Something is obviously wrong.

I am running this in a pip environment on Ubuntu 16.04. Any ideas what might be wrong here?

[alexmorley]

Hey, it looks like the sorting is generating an empty file, either due to no spikes being detected, or some other bug leading to an empty firings.mda.

Could you post the console output of the sorting stage (e.g. ml-run-process ms4alg.sort ....?

And the output of ml-spec ms4alg.sort might be useful too. Thanks! :slightly_smiling_face:

[shashwatsridhar]

Here it is :)

ml-run-process ephys.bandpass_filter --inputs timeseries:i140703-001.mda --outputs timeseries_out:i140703-001-filt.mda --parameters samplerate:30000 freq_min:300 freq_max:6000 && ml-run-process ms4alg.sort --inputs timeseries:i140703-001-filt.mda geom:utah_geom.csv --outputs firings_out:i140703-001-firings.mda --parameters adjacency_radius:0 detect_sign:-1 detect_threshold:3 clip_size:40
[ Getting processor spec... ]
(node:14880) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
[ Checking inputs and substituting prvs ... ]
[ Computing process signature ... ]
[ Checking outputs... ]
[ Checking process cache ... ]
[ Creating temporary directory ... ]
[ Initializing process ... ]
[ Running ... ] /home/sridhar/ml/bin/python3 /home/sridhar/.mountainlab/packages/ml_ephys/preprocessing/preprocessing.py ephys.bandpass_filter --timeseries=/home/sridhar/Work/Juelich/mountainlab_working/i140703-001.mda --timeseries_out=/home/sridhar/Work/Juelich/mountainlab_working/i140703-001-filt.mda --samplerate=30000 --freq_min=300 --freq_max=6000 --_tempdir=/tmp/ml/tempdir_085012768c_TScwZG
Chunk size: 30000, Padding: 3000, Num chunks: 1004, Num processes: 4

Processed 25 of 1004 chunks...
Processed 137 of 1004 chunks...
Processed 166 of 1004 chunks...
Processed 191 of 1004 chunks...
Processed 209 of 1004 chunks...
Processed 505 of 1004 chunks...
Processed 871 of 1004 chunks...
Processed 892 of 1004 chunks...
Processed 906 of 1004 chunks...

Processed 50 of 1004 chunks...
Processed 74 of 1004 chunks...
Processed 98 of 1004 chunks...
Processed 122 of 1004 chunks...
Processed 238 of 1004 chunks...
Processed 262 of 1004 chunks...
Processed 287 of 1004 chunks...
Processed 302 of 1004 chunks...
Processed 330 of 1004 chunks...
Processed 356 of 1004 chunks...
Processed 374 of 1004 chunks...
Processed 422 of 1004 chunks...
Processed 443 of 1004 chunks...
Processed 459 of 1004 chunks...
Processed 525 of 1004 chunks...
Processed 538 of 1004 chunks...
Processed 563 of 1004 chunks...
Processed 586 of 1004 chunks...
Processed 602 of 1004 chunks...

Processed 483 of 1004 chunks...
Processed 651 of 1004 chunks...
Processed 674 of 1004 chunks...
Processed 688 of 1004 chunks...
Processed 931 of 1004 chunks...
Processed 958 of 1004 chunks...
Processed 978 of 1004 chunks...

Elapsed time for processor ephys.bandpass_filter: 199.277 sec
[ Saving to process cache ... ]
[ Removing temporary directory ... ]
[ Done. ]
[ Getting processor spec... ]
(node:14997) DeprecationWarning: current URL string parser is deprecated, and will be removed in a future version. To use the new parser, pass option { useNewUrlParser: true } to MongoClient.connect.
[ Checking inputs and substituting prvs ... ]
[ Computing process signature ... ]
[ Checking outputs... ]
[ Checking process cache ... ]
[ Creating temporary directory ... ]
[ Initializing process ... ]
[ Running ... ] /home/sridhar/ml/bin/python3 /home/sridhar/.mountainlab/packages/ml_ms4alg/ms4alg_spec.py ms4alg.sort --timeseries=/home/sridhar/Work/Juelich/mountainlab_working/i140703-001-filt.mda --geom=/home/sridhar/Work/Juelich/mountainlab_working/utah_geom.csv --firings_out=/home/sridhar/Work/Juelich/mountainlab_working/i140703-001-firings.mda --adjacency_radius=0 --detect_sign=-1 --detect_threshold=3 --clip_size=40 --_tempdir=/tmp/ml/tempdir_cf48138de9_7OGgI2
Using tempdir=/tmp/ml/tempdir_cf48138de9_7OGgI2
Preparing /tmp/ml/tempdir_cf48138de9_7OGgI2/timeseries.hdf5...
Preparing neighborhood sorters...

Detecting events on channel 7 (phase1)...

Detecting events on channel 1 (phase1)...

Detecting events on channel 13 (phase1)...

Detecting events on channel 19 (phase1)...

Computing PCA features for channel 19 (phase1)...

Computing PCA features for channel 7 (phase1)...

Computing PCA features for channel 1 (phase1)...

Computing PCA features for channel 13 (phase1)...

Clustering for channel 13 (phase1)...

Clustering for channel 1 (phase1)...

Clustering for channel 7 (phase1)...

Clustering for channel 19 (phase1)...

Found 1 clusters for channel 1 (phase1)...

Computing templates for channel 1 (phase1)...

Found 1 clusters for channel 19 (phase1)...

Computing templates for channel 19 (phase1)...

Re-assigning events for channel 1 (phase1)...

Detecting events on channel 2 (phase1)...

Re-assigning events for channel 19 (phase1)...

Detecting events on channel 20 (phase1)...

Found 4 clusters for channel 13 (phase1)...
Computing templates for channel 13 (phase1)...

Computing PCA features for channel 2 (phase1)...

Computing PCA features for channel 20 (phase1)...

Re-assigning events for channel 13 (phase1)...

Detecting events on channel 14 (phase1)...

Found 7 clusters for channel 7 (phase1)...

Computing templates for channel 7 (phase1)...

Clustering for channel 2 (phase1)...

Computing PCA features for channel 14 (phase1)...

Clustering for channel 20 (phase1)...

Re-assigning events for channel 7 (phase1)...

Detecting events on channel 8 (phase1)...

Clustering for channel 14 (phase1)...

Computing PCA features for channel 8 (phase1)...

Clustering for channel 8 (phase1)...

Found 2 clusters for channel 2 (phase1)...

Computing templates for channel 2 (phase1)...

Found 2 clusters for channel 20 (phase1)...

Computing templates for channel 20 (phase1)...

Re-assigning events for channel 2 (phase1)...

Detecting events on channel 3 (phase1)...

Re-assigning events for channel 20 (phase1)...

Detecting events on channel 21 (phase1)...

Computing PCA features for channel 3 (phase1)...

Computing PCA features for channel 21 (phase1)...

Found 9 clusters for channel 14 (phase1)...

Computing templates for channel 14 (phase1)...

Clustering for channel 3 (phase1)...

Found 4 clusters for channel 8 (phase1)...

Computing templates for channel 8 (phase1)...

Clustering for channel 21 (phase1)...

Re-assigning events for channel 14 (phase1)...

Detecting events on channel 15 (phase1)...

Re-assigning events for channel 8 (phase1)...

Detecting events on channel 9 (phase1)...

Computing PCA features for channel 15 (phase1)...

Computing PCA features for channel 9 (phase1)...

Found 1 clusters for channel 3 (phase1)...
Computing templates for channel 3 (phase1)...

Clustering for channel 15 (phase1)...

Re-assigning events for channel 3 (phase1)...

Detecting events on channel 4 (phase1)...

Clustering for channel 9 (phase1)...

Computing PCA features for channel 4 (phase1)...

Found 2 clusters for channel 21 (phase1)...

Computing templates for channel 21 (phase1)...

Clustering for channel 4 (phase1)...

Found 1 clusters for channel 9 (phase1)...
Computing templates for channel 9 (phase1)...

Re-assigning events for channel 21 (phase1)...

Detecting events on channel 22 (phase1)...

Computing PCA features for channel 22 (phase1)...

Re-assigning events for channel 9 (phase1)...

Detecting events on channel 10 (phase1)...

Computing PCA features for channel 10 (phase1)...

Clustering for channel 22 (phase1)...

Found 6 clusters for channel 15 (phase1)...

Computing templates for channel 15 (phase1)...

Clustering for channel 10 (phase1)...

Re-assigning events for channel 15 (phase1)...

Detecting events on channel 16 (phase1)...

Computing PCA features for channel 16 (phase1)...

Clustering for channel 16 (phase1)...

Found 7 clusters for channel 4 (phase1)...

Computing templates for channel 4 (phase1)...

Found 1 clusters for channel 10 (phase1)...

Computing templates for channel 10 (phase1)...

Re-assigning events for channel 4 (phase1)...

Re-assigning events for channel 10 (phase1)...

Detecting events on channel 5 (phase1)...

Detecting events on channel 11 (phase1)...

Computing PCA features for channel 11 (phase1)...

Computing PCA features for channel 5 (phase1)...

Found 12 clusters for channel 22 (phase1)...
Computing templates for channel 22 (phase1)...

Re-assigning events for channel 22 (phase1)...

Detecting events on channel 23 (phase1)...

Clustering for channel 5 (phase1)...

Clustering for channel 11 (phase1)...

Computing PCA features for channel 23 (phase1)...

Clustering for channel 23 (phase1)...

Found 3 clusters for channel 16 (phase1)...
Computing templates for channel 16 (phase1)...

Found 1 clusters for channel 5 (phase1)...

Computing templates for channel 5 (phase1)...

Found 1 clusters for channel 11 (phase1)...

Computing templates for channel 11 (phase1)...

Re-assigning events for channel 16 (phase1)...

Detecting events on channel 17 (phase1)...

Re-assigning events for channel 5 (phase1)...

Detecting events on channel 6 (phase1)...

Re-assigning events for channel 11 (phase1)...

Detecting events on channel 12 (phase1)...

Computing PCA features for channel 17 (phase1)...

Computing PCA features for channel 6 (phase1)...

Computing PCA features for channel 12 (phase1)...

Found 2 clusters for channel 23 (phase1)...

Computing templates for channel 23 (phase1)...

Clustering for channel 17 (phase1)...

Clustering for channel 6 (phase1)...

Clustering for channel 12 (phase1)...

Re-assigning events for channel 23 (phase1)...

Detecting events on channel 24 (phase1)...

Computing PCA features for channel 24 (phase1)...

Found 1 clusters for channel 17 (phase1)...
Computing templates for channel 17 (phase1)...

Clustering for channel 24 (phase1)...

Re-assigning events for channel 17 (phase1)...

Detecting events on channel 18 (phase1)...

Computing PCA features for channel 18 (phase1)...

Clustering for channel 18 (phase1)...

Found 3 clusters for channel 12 (phase1)...

Computing templates for channel 12 (phase1)...

Found 7 clusters for channel 6 (phase1)...

Computing templates for channel 6 (phase1)...

Found 5 clusters for channel 24 (phase1)...

Computing templates for channel 24 (phase1)...

Re-assigning events for channel 12 (phase1)...

Detecting events on channel 25 (phase1)...

Re-assigning events for channel 6 (phase1)...

Detecting events on channel 31 (phase1)...

Re-assigning events for channel 24 (phase1)...

Detecting events on channel 37 (phase1)...

Computing PCA features for channel 25 (phase1)...

Computing PCA features for channel 31 (phase1)...

Computing PCA features for channel 37 (phase1)...

Clustering for channel 25 (phase1)...

Clustering for channel 31 (phase1)...

Clustering for channel 37 (phase1)...

Found 1 clusters for channel 25 (phase1)...
Computing templates for channel 25 (phase1)...

Found 1 clusters for channel 31 (phase1)...

Computing templates for channel 31 (phase1)...

Found 1 clusters for channel 37 (phase1)...

Computing templates for channel 37 (phase1)...

Re-assigning events for channel 25 (phase1)...

Detecting events on channel 26 (phase1)...

Re-assigning events for channel 31 (phase1)...

Detecting events on channel 32 (phase1)...

Re-assigning events for channel 37 (phase1)...

Detecting events on channel 38 (phase1)...

Computing PCA features for channel 26 (phase1)...

Computing PCA features for channel 32 (phase1)...

Found 11 clusters for channel 18 (phase1)...

Computing templates for channel 18 (phase1)...

Computing PCA features for channel 38 (phase1)...

Clustering for channel 32 (phase1)...

Clustering for channel 26 (phase1)...

Re-assigning events for channel 18 (phase1)...

Detecting events on channel 43 (phase1)...

Clustering for channel 38 (phase1)...

Computing PCA features for channel 43 (phase1)...

Clustering for channel 43 (phase1)...

Found 1 clusters for channel 38 (phase1)...

Computing templates for channel 38 (phase1)...

Found 1 clusters for channel 43 (phase1)...

Computing templates for channel 43 (phase1)...

Re-assigning events for channel 38 (phase1)...

Detecting events on channel 39 (phase1)...

Re-assigning events for channel 43 (phase1)...

Detecting events on channel 44 (phase1)...

Computing PCA features for channel 39 (phase1)...

Found 3 clusters for channel 26 (phase1)...
Computing templates for channel 26 (phase1)...

Found 4 clusters for channel 32 (phase1)...

Computing templates for channel 32 (phase1)...

Computing PCA features for channel 44 (phase1)...

Re-assigning events for channel 26 (phase1)...

Detecting events on channel 27 (phase1)...

Clustering for channel 39 (phase1)...

Re-assigning events for channel 32 (phase1)...

Detecting events on channel 33 (phase1)...

Clustering for channel 44 (phase1)...

Computing PCA features for channel 27 (phase1)...

Computing PCA features for channel 33 (phase1)...

Found 1 clusters for channel 39 (phase1)...

Computing templates for channel 39 (phase1)...

Clustering for channel 27 (phase1)...

Found 1 clusters for channel 44 (phase1)...
Computing templates for channel 44 (phase1)...

Clustering for channel 33 (phase1)...

Re-assigning events for channel 39 (phase1)...

Detecting events on channel 40 (phase1)...

Re-assigning events for channel 44 (phase1)...

Detecting events on channel 45 (phase1)...

Computing PCA features for channel 40 (phase1)...

Computing PCA features for channel 45 (phase1)...

Clustering for channel 40 (phase1)...

Clustering for channel 45 (phase1)...

Found 1 clusters for channel 45 (phase1)...

Computing templates for channel 45 (phase1)...

Found 3 clusters for channel 33 (phase1)...

Computing templates for channel 33 (phase1)...

Found 6 clusters for channel 27 (phase1)...

Computing templates for channel 27 (phase1)...

Re-assigning events for channel 45 (phase1)...

Detecting events on channel 46 (phase1)...

Re-assigning events for channel 33 (phase1)...

Detecting events on channel 34 (phase1)...

Re-assigning events for channel 27 (phase1)...

Detecting events on channel 28 (phase1)...

Computing PCA features for channel 46 (phase1)...

Computing PCA features for channel 34 (phase1)...

Found 3 clusters for channel 40 (phase1)...

Computing templates for channel 40 (phase1)...

Computing PCA features for channel 28 (phase1)...

Clustering for channel 46 (phase1)...

Clustering for channel 34 (phase1)...

Re-assigning events for channel 40 (phase1)...

Detecting events on channel 41 (phase1)...

Clustering for channel 28 (phase1)...

Computing PCA features for channel 41 (phase1)...

Found 1 clusters for channel 46 (phase1)...
Computing templates for channel 46 (phase1)...

Found 1 clusters for channel 34 (phase1)...
Computing templates for channel 34 (phase1)...

Clustering for channel 41 (phase1)...

Re-assigning events for channel 46 (phase1)...

Detecting events on channel 47 (phase1)...

Re-assigning events for channel 34 (phase1)...

Detecting events on channel 35 (phase1)...

Computing PCA features for channel 47 (phase1)...

Computing PCA features for channel 35 (phase1)...

Clustering for channel 47 (phase1)...

Clustering for channel 35 (phase1)...

Found 3 clusters for channel 28 (phase1)...
Computing templates for channel 28 (phase1)...

Found 1 clusters for channel 35 (phase1)...
Computing templates for channel 35 (phase1)...

Re-assigning events for channel 28 (phase1)...

Detecting events on channel 29 (phase1)...

Found 1 clusters for channel 47 (phase1)...
Computing templates for channel 47 (phase1)...

Computing PCA features for channel 29 (phase1)...

Found 5 clusters for channel 41 (phase1)...

Computing templates for channel 41 (phase1)...

Re-assigning events for channel 35 (phase1)...

Detecting events on channel 36 (phase1)...

Re-assigning events for channel 47 (phase1)...

Detecting events on channel 48 (phase1)...

Clustering for channel 29 (phase1)...

Computing PCA features for channel 36 (phase1)...

Re-assigning events for channel 41 (phase1)...

Detecting events on channel 42 (phase1)...

Computing PCA features for channel 48 (phase1)...

Computing PCA features for channel 42 (phase1)...

Clustering for channel 36 (phase1)...

Clustering for channel 48 (phase1)...

Clustering for channel 42 (phase1)...

Found 2 clusters for channel 29 (phase1)...
Computing templates for channel 29 (phase1)...

Re-assigning events for channel 29 (phase1)...

Detecting events on channel 30 (phase1)...

Found 7 clusters for channel 36 (phase1)...

Computing templates for channel 36 (phase1)...

Computing PCA features for channel 30 (phase1)...

Found 16 clusters for channel 48 (phase1)...

Computing templates for channel 48 (phase1)...

Re-assigning events for channel 36 (phase1)...

Detecting events on channel 49 (phase1)...

Clustering for channel 30 (phase1)...

Computing PCA features for channel 49 (phase1)...

Re-assigning events for channel 48 (phase1)...

Detecting events on channel 55 (phase1)...

Found 6 clusters for channel 42 (phase1)...

Computing templates for channel 42 (phase1)...

Computing PCA features for channel 55 (phase1)...

Clustering for channel 49 (phase1)...

Re-assigning events for channel 42 (phase1)...

Detecting events on channel 61 (phase1)...

Clustering for channel 55 (phase1)...

Computing PCA features for channel 61 (phase1)...

Clustering for channel 61 (phase1)...

Found 3 clusters for channel 30 (phase1)...

Computing templates for channel 30 (phase1)...

Found 7 clusters for channel 49 (phase1)...

Computing templates for channel 49 (phase1)...

Re-assigning events for channel 30 (phase1)...

Detecting events on channel 67 (phase1)...

Found 2 clusters for channel 55 (phase1)...
Computing templates for channel 55 (phase1)...

Computing PCA features for channel 67 (phase1)...

Re-assigning events for channel 49 (phase1)...

Detecting events on channel 50 (phase1)...

Re-assigning events for channel 55 (phase1)...

Detecting events on channel 56 (phase1)...

Found 3 clusters for channel 61 (phase1)...

Computing templates for channel 61 (phase1)...

Computing PCA features for channel 50 (phase1)...

Clustering for channel 67 (phase1)...

Computing PCA features for channel 56 (phase1)...

Re-assigning events for channel 61 (phase1)...

Detecting events on channel 62 (phase1)...

Clustering for channel 50 (phase1)...

Computing PCA features for channel 62 (phase1)...

Clustering for channel 56 (phase1)...

Found 1 clusters for channel 67 (phase1)...

Computing templates for channel 67 (phase1)...

Clustering for channel 62 (phase1)...

Re-assigning events for channel 67 (phase1)...

Detecting events on channel 68 (phase1)...

Computing PCA features for channel 68 (phase1)...

Clustering for channel 68 (phase1)...

Found 8 clusters for channel 50 (phase1)...
Computing templates for channel 50 (phase1)...

Found 8 clusters for channel 56 (phase1)...

Computing templates for channel 56 (phase1)...

Re-assigning events for channel 50 (phase1)...

Detecting events on channel 51 (phase1)...

Re-assigning events for channel 56 (phase1)...

Detecting events on channel 57 (phase1)...

Computing PCA features for channel 51 (phase1)...

Computing PCA features for channel 57 (phase1)...

Found 3 clusters for channel 62 (phase1)...

Computing templates for channel 62 (phase1)...

Found 1 clusters for channel 68 (phase1)...

Computing templates for channel 68 (phase1)...

Clustering for channel 51 (phase1)...

Re-assigning events for channel 62 (phase1)...

Detecting events on channel 63 (phase1)...

Clustering for channel 57 (phase1)...

Re-assigning events for channel 68 (phase1)...

Detecting events on channel 69 (phase1)...

Computing PCA features for channel 63 (phase1)...

Computing PCA features for channel 69 (phase1)...

Found 1 clusters for channel 51 (phase1)...
Computing templates for channel 51 (phase1)...

Found 1 clusters for channel 57 (phase1)...

Computing templates for channel 57 (phase1)...

Clustering for channel 63 (phase1)...

Clustering for channel 69 (phase1)...

Re-assigning events for channel 51 (phase1)...

Detecting events on channel 52 (phase1)...

Re-assigning events for channel 57 (phase1)...

Detecting events on channel 58 (phase1)...

Computing PCA features for channel 52 (phase1)...

Computing PCA features for channel 58 (phase1)...

Clustering for channel 52 (phase1)...

Clustering for channel 58 (phase1)...

Found 1 clusters for channel 69 (phase1)...

Computing templates for channel 69 (phase1)...

Re-assigning events for channel 69 (phase1)...

Detecting events on channel 70 (phase1)...

Computing PCA features for channel 70 (phase1)...

Found 1 clusters for channel 58 (phase1)...
Computing templates for channel 58 (phase1)...

Re-assigning events for channel 58 (phase1)...

Detecting events on channel 59 (phase1)...

Clustering for channel 70 (phase1)...

Computing PCA features for channel 59 (phase1)...

Found 18 clusters for channel 63 (phase1)...

Computing templates for channel 63 (phase1)...

Found 1 clusters for channel 70 (phase1)...

Computing templates for channel 70 (phase1)...

Clustering for channel 59 (phase1)...

Re-assigning events for channel 63 (phase1)...

Detecting events on channel 64 (phase1)...

Re-assigning events for channel 70 (phase1)...

Detecting events on channel 71 (phase1)...

Computing PCA features for channel 64 (phase1)...

Computing PCA features for channel 71 (phase1)...

Clustering for channel 64 (phase1)...

Found 19 clusters for channel 52 (phase1)...

Computing templates for channel 52 (phase1)...

Clustering for channel 71 (phase1)...

Re-assigning events for channel 52 (phase1)...

Detecting events on channel 53 (phase1)...

Found 1 clusters for channel 64 (phase1)...

Computing templates for channel 64 (phase1)...

Computing PCA features for channel 53 (phase1)...

Re-assigning events for channel 64 (phase1)...

Detecting events on channel 65 (phase1)...

Found 10 clusters for channel 59 (phase1)...

Computing templates for channel 59 (phase1)...

Clustering for channel 53 (phase1)...

Computing PCA features for channel 65 (phase1)...

Re-assigning events for channel 59 (phase1)...

Detecting events on channel 60 (phase1)...

Clustering for channel 65 (phase1)...

Computing PCA features for channel 60 (phase1)...

Found 5 clusters for channel 71 (phase1)...

Computing templates for channel 71 (phase1)...

Found 1 clusters for channel 53 (phase1)...
Computing templates for channel 53 (phase1)...

Clustering for channel 60 (phase1)...

Re-assigning events for channel 71 (phase1)...

Detecting events on channel 72 (phase1)...

Re-assigning events for channel 53 (phase1)...

Detecting events on channel 54 (phase1)...

Computing PCA features for channel 72 (phase1)...

Found 1 clusters for channel 65 (phase1)...

Computing templates for channel 65 (phase1)...

Computing PCA features for channel 54 (phase1)...

Clustering for channel 72 (phase1)...

Re-assigning events for channel 65 (phase1)...

Detecting events on channel 66 (phase1)...

Clustering for channel 54 (phase1)...

Computing PCA features for channel 66 (phase1)...

Found 1 clusters for channel 72 (phase1)...

Computing templates for channel 72 (phase1)...

Found 4 clusters for channel 60 (phase1)...
Computing templates for channel 60 (phase1)...

Found 1 clusters for channel 54 (phase1)...

Computing templates for channel 54 (phase1)...

Clustering for channel 66 (phase1)...

Re-assigning events for channel 72 (phase1)...

Re-assigning events for channel 60 (phase1)...

Detecting events on channel 73 (phase1)...

Detecting events on channel 79 (phase1)...

Re-assigning events for channel 54 (phase1)...

Detecting events on channel 85 (phase1)...

Computing PCA features for channel 73 (phase1)...

Computing PCA features for channel 79 (phase1)...

Computing PCA features for channel 85 (phase1)...

Clustering for channel 73 (phase1)...

Clustering for channel 79 (phase1)...

Clustering for channel 85 (phase1)...

Found 9 clusters for channel 66 (phase1)...

Computing templates for channel 66 (phase1)...

Found 6 clusters for channel 79 (phase1)...
Computing templates for channel 79 (phase1)...

Found 10 clusters for channel 85 (phase1)...

Computing templates for channel 85 (phase1)...

Found 19 clusters for channel 73 (phase1)...

Computing templates for channel 73 (phase1)...

Re-assigning events for channel 66 (phase1)...

Detecting events on channel 91 (phase1)...

Re-assigning events for channel 79 (phase1)...

Detecting events on channel 80 (phase1)...

Re-assigning events for channel 85 (phase1)...

Detecting events on channel 86 (phase1)...

Computing PCA features for channel 91 (phase1)...

Re-assigning events for channel 73 (phase1)...

Detecting events on channel 74 (phase1)...

Computing PCA features for channel 80 (phase1)...

Computing PCA features for channel 86 (phase1)...

Computing PCA features for channel 74 (phase1)...

Clustering for channel 91 (phase1)...

Clustering for channel 80 (phase1)...

Clustering for channel 86 (phase1)...

Clustering for channel 74 (phase1)...

Found 1 clusters for channel 80 (phase1)...

Computing templates for channel 80 (phase1)...

Re-assigning events for channel 80 (phase1)...

Detecting events on channel 81 (phase1)...

Computing PCA features for channel 81 (phase1)...

Found 4 clusters for channel 91 (phase1)...

Computing templates for channel 91 (phase1)...

Clustering for channel 81 (phase1)...

Found 3 clusters for channel 86 (phase1)...
Computing templates for channel 86 (phase1)...

Found 2 clusters for channel 74 (phase1)...

Computing templates for channel 74 (phase1)...

Re-assigning events for channel 91 (phase1)...

Detecting events on channel 92 (phase1)...

Re-assigning events for channel 86 (phase1)...

Detecting events on channel 87 (phase1)...

Computing PCA features for channel 92 (phase1)...

Re-assigning events for channel 74 (phase1)...

Detecting events on channel 75 (phase1)...

Computing PCA features for channel 87 (phase1)...

Computing PCA features for channel 75 (phase1)...

Clustering for channel 92 (phase1)...

Clustering for channel 87 (phase1)...

Clustering for channel 75 (phase1)...

Found 1 clusters for channel 87 (phase1)...
Computing templates for channel 87 (phase1)...

Found 5 clusters for channel 81 (phase1)...

Computing templates for channel 81 (phase1)...

Re-assigning events for channel 87 (phase1)...

Detecting events on channel 88 (phase1)...

Re-assigning events for channel 81 (phase1)...

Computing PCA features for channel 88 (phase1)...

Detecting events on channel 82 (phase1)...

Found 3 clusters for channel 92 (phase1)...

Computing templates for channel 92 (phase1)...

Computing PCA features for channel 82 (phase1)...

Clustering for channel 88 (phase1)...

Re-assigning events for channel 92 (phase1)...

Detecting events on channel 93 (phase1)...

Found 7 clusters for channel 75 (phase1)...
Computing templates for channel 75 (phase1)...

Clustering for channel 82 (phase1)...

Computing PCA features for channel 93 (phase1)...

Re-assigning events for channel 75 (phase1)...

Detecting events on channel 76 (phase1)...

Computing PCA features for channel 76 (phase1)...

Clustering for channel 93 (phase1)...

Found 1 clusters for channel 88 (phase1)...

Computing templates for channel 88 (phase1)...

Found 1 clusters for channel 82 (phase1)...
Computing templates for channel 82 (phase1)...

Clustering for channel 76 (phase1)...

Re-assigning events for channel 88 (phase1)...

Detecting events on channel 89 (phase1)...

Re-assigning events for channel 82 (phase1)...

Detecting events on channel 83 (phase1)...

Computing PCA features for channel 89 (phase1)...

Found 1 clusters for channel 93 (phase1)...

Computing templates for channel 93 (phase1)...

Computing PCA features for channel 83 (phase1)...

Found 1 clusters for channel 76 (phase1)...
Computing templates for channel 76 (phase1)...

Re-assigning events for channel 93 (phase1)...

Detecting events on channel 94 (phase1)...

Clustering for channel 83 (phase1)...

Clustering for channel 89 (phase1)...

Re-assigning events for channel 76 (phase1)...

Detecting events on channel 77 (phase1)...

Computing PCA features for channel 94 (phase1)...

Computing PCA features for channel 77 (phase1)...

Found 1 clusters for channel 89 (phase1)...

Computing templates for channel 89 (phase1)...

Clustering for channel 94 (phase1)...

Clustering for channel 77 (phase1)...

Re-assigning events for channel 89 (phase1)...

Detecting events on channel 90 (phase1)...

Computing PCA features for channel 90 (phase1)...

Found 7 clusters for channel 83 (phase1)...

Computing templates for channel 83 (phase1)...

Clustering for channel 90 (phase1)...

Re-assigning events for channel 83 (phase1)...

Detecting events on channel 84 (phase1)...

Computing PCA features for channel 84 (phase1)...

Found 3 clusters for channel 77 (phase1)...
Computing templates for channel 77 (phase1)...

Found 1 clusters for channel 90 (phase1)...

Computing templates for channel 90 (phase1)...

Found 4 clusters for channel 94 (phase1)...

Computing templates for channel 94 (phase1)...

Clustering for channel 84 (phase1)...

Re-assigning events for channel 77 (phase1)...

Detecting events on channel 78 (phase1)...

Re-assigning events for channel 90 (phase1)...

Re-assigning events for channel 94 (phase1)...

Detecting events on channel 95 (phase1)...

Computing PCA features for channel 78 (phase1)...

Computing PCA features for channel 95 (phase1)...

Found 1 clusters for channel 84 (phase1)...

Computing templates for channel 84 (phase1)...

Clustering for channel 78 (phase1)...

Clustering for channel 95 (phase1)...

Re-assigning events for channel 84 (phase1)...

Found 4 clusters for channel 78 (phase1)...

Computing templates for channel 78 (phase1)...

Re-assigning events for channel 78 (phase1)...

Found 8 clusters for channel 95 (phase1)...
Computing templates for channel 95 (phase1)...

Re-assigning events for channel 95 (phase1)...

Detecting events on channel 96 (phase1)...

Computing PCA features for channel 96 (phase1)...

Clustering for channel 96 (phase1)...

Found 3 clusters for channel 96 (phase1)...
Computing templates for channel 96 (phase1)...

Re-assigning events for channel 96 (phase1)...

Computing PCA features for channel 7 (phase2)...
Computing PCA features for channel 13 (phase2)...
Computing PCA features for channel 1 (phase2)...

Computing PCA features for channel 19 (phase2)...

Clustering for channel 13 (phase2)...

Clustering for channel 1 (phase2)...

Clustering for channel 7 (phase2)...

Clustering for channel 19 (phase2)...

Found 1 clusters for channel 1 (phase2)...

Computing PCA features for channel 2 (phase2)...

Found 1 clusters for channel 19 (phase2)...

Computing PCA features for channel 20 (phase2)...

Clustering for channel 20 (phase2)...

Clustering for channel 2 (phase2)...

Found 4 clusters for channel 13 (phase2)...

Computing PCA features for channel 14 (phase2)...

Found 5 clusters for channel 7 (phase2)...

Computing PCA features for channel 8 (phase2)...

Clustering for channel 14 (phase2)...

Clustering for channel 8 (phase2)...

Found 2 clusters for channel 2 (phase2)...

Computing PCA features for channel 3 (phase2)...

Found 4 clusters for channel 20 (phase2)...

Computing PCA features for channel 21 (phase2)...

Clustering for channel 3 (phase2)...

Clustering for channel 21 (phase2)...

Found 1 clusters for channel 3 (phase2)...

Computing PCA features for channel 4 (phase2)...

Found 8 clusters for channel 14 (phase2)...

Computing PCA features for channel 15 (phase2)...

Found 8 clusters for channel 8 (phase2)...

Computing PCA features for channel 9 (phase2)...

Found 1 clusters for channel 21 (phase2)...

Computing PCA features for channel 22 (phase2)...

Clustering for channel 4 (phase2)...

Clustering for channel 15 (phase2)...

Clustering for channel 9 (phase2)...

Clustering for channel 22 (phase2)...

Found 1 clusters for channel 9 (phase2)...

Computing PCA features for channel 10 (phase2)...

Clustering for channel 10 (phase2)...

Found 5 clusters for channel 4 (phase2)...

Computing PCA features for channel 5 (phase2)...

Found 7 clusters for channel 15 (phase2)...

Computing PCA features for channel 16 (phase2)...

Found 1 clusters for channel 10 (phase2)...

Computing PCA features for channel 11 (phase2)...

Clustering for channel 5 (phase2)...

Clustering for channel 16 (phase2)...

Clustering for channel 11 (phase2)...

Found 8 clusters for channel 22 (phase2)...

Computing PCA features for channel 23 (phase2)...

Found 1 clusters for channel 5 (phase2)...

Computing PCA features for channel 6 (phase2)...

Clustering for channel 23 (phase2)...

Found 1 clusters for channel 11 (phase2)...

Computing PCA features for channel 12 (phase2)...

Clustering for channel 6 (phase2)...

Clustering for channel 12 (phase2)...

Found 3 clusters for channel 16 (phase2)...

Computing PCA features for channel 17 (phase2)...

Found 2 clusters for channel 23 (phase2)...

Computing PCA features for channel 24 (phase2)...

Clustering for channel 17 (phase2)...

Clustering for channel 24 (phase2)...

Found 5 clusters for channel 6 (phase2)...

Computing PCA features for channel 25 (phase2)...

Found 1 clusters for channel 17 (phase2)...

Computing PCA features for channel 18 (phase2)...

Clustering for channel 25 (phase2)...

Found 3 clusters for channel 12 (phase2)...

Computing PCA features for channel 31 (phase2)...

Clustering for channel 18 (phase2)...

Clustering for channel 31 (phase2)...

Found 1 clusters for channel 25 (phase2)...

Computing PCA features for channel 26 (phase2)...

Found 3 clusters for channel 24 (phase2)...

Computing PCA features for channel 37 (phase2)...

Found 1 clusters for channel 31 (phase2)...

Computing PCA features for channel 32 (phase2)...

Clustering for channel 26 (phase2)...

Clustering for channel 37 (phase2)...

Clustering for channel 32 (phase2)...

Found 1 clusters for channel 37 (phase2)...

Computing PCA features for channel 38 (phase2)...

Clustering for channel 38 (phase2)...

Found 5 clusters for channel 18 (phase2)...

Computing PCA features for channel 43 (phase2)...

Found 2 clusters for channel 26 (phase2)...

Computing PCA features for channel 27 (phase2)...

Clustering for channel 43 (phase2)...

Clustering for channel 27 (phase2)...

Found 1 clusters for channel 43 (phase2)...

Computing PCA features for channel 44 (phase2)...

Found 6 clusters for channel 32 (phase2)...

Computing PCA features for channel 33 (phase2)...

Clustering for channel 44 (phase2)...

Found 2 clusters for channel 38 (phase2)...

Computing PCA features for channel 39 (phase2)...

Clustering for channel 33 (phase2)...

Clustering for channel 39 (phase2)...

Found 1 clusters for channel 44 (phase2)...

Computing PCA features for channel 45 (phase2)...

Found 4 clusters for channel 27 (phase2)...

Computing PCA features for channel 28 (phase2)...

Clustering for channel 45 (phase2)...

Found 1 clusters for channel 39 (phase2)...

Computing PCA features for channel 40 (phase2)...

Clustering for channel 28 (phase2)...

Clustering for channel 40 (phase2)...

Found 1 clusters for channel 45 (phase2)...

Computing PCA features for channel 46 (phase2)...

Clustering for channel 46 (phase2)...

Found 3 clusters for channel 33 (phase2)...

Computing PCA features for channel 34 (phase2)...

Clustering for channel 34 (phase2)...

Found 1 clusters for channel 46 (phase2)...

Computing PCA features for channel 47 (phase2)...

Found 4 clusters for channel 28 (phase2)...

Computing PCA features for channel 29 (phase2)...

Found 2 clusters for channel 40 (phase2)...

Computing PCA features for channel 41 (phase2)...

Clustering for channel 47 (phase2)...

Clustering for channel 29 (phase2)...

Found 1 clusters for channel 34 (phase2)...

Computing PCA features for channel 35 (phase2)...

Clustering for channel 41 (phase2)...

Clustering for channel 35 (phase2)...

Found 1 clusters for channel 47 (phase2)...

Computing PCA features for channel 48 (phase2)...

Found 1 clusters for channel 35 (phase2)...

Computing PCA features for channel 36 (phase2)...

Clustering for channel 48 (phase2)...

Clustering for channel 36 (phase2)...

Found 3 clusters for channel 29 (phase2)...

Computing PCA features for channel 30 (phase2)...

Found 3 clusters for channel 41 (phase2)...

Computing PCA features for channel 42 (phase2)...

Clustering for channel 30 (phase2)...

Clustering for channel 42 (phase2)...

Found 11 clusters for channel 48 (phase2)...

Computing PCA features for channel 49 (phase2)...

Found 7 clusters for channel 36 (phase2)...

Computing PCA features for channel 55 (phase2)...

Clustering for channel 49 (phase2)...

Clustering for channel 55 (phase2)...

Found 4 clusters for channel 30 (phase2)...

Computing PCA features for channel 61 (phase2)...

Found 3 clusters for channel 42 (phase2)...

Computing PCA features for channel 67 (phase2)...

Clustering for channel 61 (phase2)...

Clustering for channel 67 (phase2)...

Found 4 clusters for channel 49 (phase2)...

Computing PCA features for channel 50 (phase2)...

Found 2 clusters for channel 55 (phase2)...

Computing PCA features for channel 56 (phase2)...

Clustering for channel 50 (phase2)...

Clustering for channel 56 (phase2)...

Found 3 clusters for channel 61 (phase2)...

Computing PCA features for channel 62 (phase2)...

Found 3 clusters for channel 67 (phase2)...

Computing PCA features for channel 68 (phase2)...

Clustering for channel 62 (phase2)...

Clustering for channel 68 (phase2)...

Found 6 clusters for channel 50 (phase2)...

Computing PCA features for channel 51 (phase2)...

Found 6 clusters for channel 56 (phase2)...

Computing PCA features for channel 57 (phase2)...

Found 1 clusters for channel 68 (phase2)...

Computing PCA features for channel 69 (phase2)...

Found 3 clusters for channel 62 (phase2)...

Computing PCA features for channel 63 (phase2)...

Clustering for channel 51 (phase2)...

Clustering for channel 57 (phase2)...

Clustering for channel 69 (phase2)...

Clustering for channel 63 (phase2)...

Found 1 clusters for channel 51 (phase2)...

Computing PCA features for channel 52 (phase2)...

Found 1 clusters for channel 57 (phase2)...

Computing PCA features for channel 58 (phase2)...

Found 1 clusters for channel 69 (phase2)...

Computing PCA features for channel 70 (phase2)...

Clustering for channel 52 (phase2)...

Clustering for channel 58 (phase2)...

Clustering for channel 70 (phase2)...

Found 1 clusters for channel 58 (phase2)...

Computing PCA features for channel 59 (phase2)...

Found 1 clusters for channel 70 (phase2)...

Computing PCA features for channel 71 (phase2)...

Found 15 clusters for channel 63 (phase2)...

Computing PCA features for channel 64 (phase2)...

Clustering for channel 71 (phase2)...

Clustering for channel 59 (phase2)...

Clustering for channel 64 (phase2)...

Found 1 clusters for channel 64 (phase2)...

Computing PCA features for channel 65 (phase2)...

Found 14 clusters for channel 52 (phase2)...

Computing PCA features for channel 53 (phase2)...

Clustering for channel 65 (phase2)...

Found 4 clusters for channel 71 (phase2)...

Computing PCA features for channel 72 (phase2)...

Clustering for channel 53 (phase2)...

Clustering for channel 72 (phase2)...

Found 8 clusters for channel 59 (phase2)...

Computing PCA features for channel 60 (phase2)...

Found 1 clusters for channel 65 (phase2)...

Computing PCA features for channel 66 (phase2)...

Clustering for channel 60 (phase2)...

Found 1 clusters for channel 72 (phase2)...

Computing PCA features for channel 73 (phase2)...

Clustering for channel 66 (phase2)...

Found 1 clusters for channel 53 (phase2)...

Computing PCA features for channel 54 (phase2)...

Found 1 clusters for channel 60 (phase2)...

Computing PCA features for channel 79 (phase2)...

Clustering for channel 73 (phase2)...

Clustering for channel 54 (phase2)...

Clustering for channel 79 (phase2)...

Found 1 clusters for channel 54 (phase2)...

Computing PCA features for channel 85 (phase2)...

Clustering for channel 85 (phase2)...

Found 4 clusters for channel 79 (phase2)...

Computing PCA features for channel 80 (phase2)...

Found 12 clusters for channel 73 (phase2)...

Computing PCA features for channel 74 (phase2)...

Found 8 clusters for channel 66 (phase2)...

Computing PCA features for channel 91 (phase2)...

Clustering for channel 80 (phase2)...

Clustering for channel 91 (phase2)...

Clustering for channel 74 (phase2)...

Found 6 clusters for channel 85 (phase2)...

Computing PCA features for channel 86 (phase2)...

Found 1 clusters for channel 80 (phase2)...

Computing PCA features for channel 81 (phase2)...

Clustering for channel 86 (phase2)...

Clustering for channel 81 (phase2)...

Found 2 clusters for channel 74 (phase2)...

Computing PCA features for channel 75 (phase2)...

Found 4 clusters for channel 91 (phase2)...

Computing PCA features for channel 92 (phase2)...

Clustering for channel 75 (phase2)...

Found 3 clusters for channel 86 (phase2)...

Computing PCA features for channel 87 (phase2)...

Clustering for channel 92 (phase2)...

Clustering for channel 87 (phase2)...

Found 5 clusters for channel 81 (phase2)...

Computing PCA features for channel 82 (phase2)...

Found 1 clusters for channel 87 (phase2)...

Computing PCA features for channel 88 (phase2)...

Found 3 clusters for channel 92 (phase2)...

Computing PCA features for channel 93 (phase2)...

Clustering for channel 82 (phase2)...

Found 8 clusters for channel 75 (phase2)...

Computing PCA features for channel 76 (phase2)...

Clustering for channel 88 (phase2)...

Clustering for channel 93 (phase2)...

Clustering for channel 76 (phase2)...

Found 1 clusters for channel 82 (phase2)...

Computing PCA features for channel 83 (phase2)...

Found 1 clusters for channel 76 (phase2)...

Found 1 clusters for channel 88 (phase2)...

Computing PCA features for channel 77 (phase2)...

Computing PCA features for channel 89 (phase2)...

Clustering for channel 83 (phase2)...

Found 1 clusters for channel 93 (phase2)...

Computing PCA features for channel 94 (phase2)...

Clustering for channel 89 (phase2)...

Clustering for channel 77 (phase2)...

Clustering for channel 94 (phase2)...

Found 1 clusters for channel 89 (phase2)...

Computing PCA features for channel 90 (phase2)...

Clustering for channel 90 (phase2)...

Found 7 clusters for channel 83 (phase2)...

Computing PCA features for channel 84 (phase2)...

Found 3 clusters for channel 77 (phase2)...

Computing PCA features for channel 78 (phase2)...

Clustering for channel 84 (phase2)...

Found 1 clusters for channel 90 (phase2)...

Found 4 clusters for channel 94 (phase2)...

Computing PCA features for channel 95 (phase2)...

Clustering for channel 78 (phase2)...

Found 1 clusters for channel 84 (phase2)...

Clustering for channel 95 (phase2)...

Found 3 clusters for channel 78 (phase2)...

Found 7 clusters for channel 95 (phase2)...

Computing PCA features for channel 96 (phase2)...

Clustering for channel 96 (phase2)...

Found 3 clusters for channel 96 (phase2)...

Preparing output...

Writing firings file...

Done.

Thanks for the quick reply!

[magland]

Could you confirm the size of the file: /home/sridhar/Work/Juelich/mountainlab_working/i140703-001-firings.mda

[shashwatsridhar]

20 bytes!

[shashwatsridhar]

I also notice now how the modified doesn't reflect today's date, instead shows yesterday's date. That is strange.

[magland]

That's strange. could you remove the output file... and run again to make sure it's getting created?

[shashwatsridhar]

Will do and let you know!

[shashwatsridhar]

In the meantime, the output for ml-spec ms4alg.sort:

{
    "description": "MountainSort spike sorting (version 4)",
    "exe_command": "/home/sridhar/ml/bin/python3 /home/sridhar/.mountainlab/packages/ml_ms4alg/ms4alg_spec.py ms4alg.sort $(arguments)",
    "inputs": [
        {
            "description": "MxN raw timeseries array (M = #channels, N = #timepoints)",
            "name": "timeseries",
            "optional": false
        },
        {
            "default_value": "",
            "description": "Optional geometry file (.csv format)",
            "name": "geom",
            "optional": true
        }
    ],
    "name": "ms4alg.sort",
    "outputs": [
        {
            "description": "Firings array channels/times/labels (3xL, L = num. events)",
            "name": "firings_out",
            "optional": false
        }
    ],
    "parameters": [
        {
            "datatype": "float",
            "description": "Radius of local sorting neighborhood, corresponding to the geometry file (same units). 0 means each channel is sorted independently. -1 means all channels are included in every neighborhood.",
            "name": "adjacency_radius",
            "optional": false
        },
        {
            "datatype": "int",
            "description": "Use 1, -1, or 0 to detect positive peaks, negative peaks, or both, respectively",
            "name": "detect_sign",
            "optional": false
        },
        {
            "datatype": "float",
            "default_value": 3,
            "description": "Threshold for event detection, corresponding to the input file. So if the input file is normalized to have noise standard deviation 1 (e.g., whitened), then this is in units of std. deviations away from the mean.",
            "name": "detect_threshold",
            "optional": true
        },
        {
            "datatype": "int",
            "default_value": 10,
            "description": "The minimum number of timepoints between adjacent spikes detected in the same channel neighborhood.",
            "name": "detect_interval",
            "optional": true
        },
        {
            "datatype": "int",
            "default_value": 50,
            "description": "Size of extracted clips or snippets, used throughout",
            "name": "clip_size",
            "optional": true
        },
        {
            "datatype": "int",
            "default_value": 4,
            "description": "Number of simultaneous workers (or processes). The default is multiprocessing.cpu_count().",
            "name": "num_workers",
            "optional": true
        }
    ],
    "version": "0.11"
}

[shashwatsridhar]

So, I ran again, and the result is the same. A 20byte firings.mda file.

Any hints what might be wrong?

[magland]

Strange. The only thing that I can think of is that there were no events found. But that doesn't seem right considering that it reports finding clusters.

Would you be able to help with debugging? Are you familiar with python? Here's the relevant file: https://github.com/magland/ml_ms4alg/blob/master/ms4alg.py

Note at the bottom where it says "preparing output..." (see below). It would be helpful to put in some debug statements, like to report len(labels) for each channel. Could it really be empty?

        print ('Preparing output...'); sys.stdout.flush()
        all_times_list=[]
        all_labels_list=[]
        all_channels_list=[]
        k_offset=0
        for m in range(M):
            labels=neighborhood_sorters[m].getPhase2Labels()
            all_times_list.append(neighborhood_sorters[m].getPhase2Times())
            all_labels_list.append(labels+k_offset)
            all_channels_list.append(np.ones(len(neighborhood_sorters[m].getPhase2Times()))*(m+1))
            k_offset+=np.max(labels) if labels.size > 0 else 0

        all_times=np.concatenate(all_times_list)
        all_labels=np.concatenate(all_labels_list)
        all_channels=np.concatenate(all_channels_list)

        sort_inds=np.argsort(all_times)
        all_times=all_times[sort_inds]
        all_labels=all_labels[sort_inds]
        all_channels=all_channels[sort_inds]

        print ('Writing firings file...'); sys.stdout.flush()
        write_firings_file(all_channels,all_times,all_labels,self._firings_out_path)

        print ('Done.'); sys.stdout.flush()

[shashwatsridhar]

So, I figured out where the error was occurring. The write_firings_file function at the end was calling mdaio.writemda64 to create the final firings file. I changed it to writemda32 and also changed the dtype of the X array to float32. The output firings.mda file is now a sensible 1.3Gb. I am not well versed with the details of memory management in python, but I have a feeling my laptop couldn't handle such a large amount of 64bit data. I am running an Ubuntu 16.04 system on Intel Core i5-2520M (2.50GHz) with 8Gb of RAM. I am pasting the last few lines of the console output here, in case it interests you.

...

Found 1 clusters for channel 84 (phase2)...

Found 1 clusters for channel 90 (phase2)...

Found 4 clusters for channel 94 (phase2)...

Computing PCA features for channel 95 (phase2)...

Found 3 clusters for channel 78 (phase2)...

Clustering for channel 95 (phase2)...

Found 5 clusters for channel 95 (phase2)...

Computing PCA features for channel 96 (phase2)...

Clustering for channel 96 (phase2)...

Found 3 clusters for channel 96 (phase2)...

Preparing output...

Length of all_times: 109122726

Length of all_labels: 109122726

Length of all_channels: 109122726

Writing firings file...

Random parts of channels: [73.  5.  5. 54. 29. 60. 88. 36. 46. 65.  2. 84. 37. 28. 76. 16. 18. 50.
  3. 48. 86. 71.  9. 94.  8. 26. 36. 60. 30. 68. 65. 50. 64. 26. 43. 29.
 31. 91. 28. 95. 17. 54. 91. 83. 86. 95. 41. 26. 49. 88. 83. 17. 87. 11.
 93. 41. 50. 62. 34. 19. 57.  1. 18. 25. 12. 12. 66. 33. 31. 54. 21. 12.
 90. 48. 54. 81. 71.  5.  4.  9. 26. 10. 49.  7. 56. 96. 11. 10. 47.  5.
 82. 37. 25. 33. 83. 36. 50. 68. 28. 60.]

Random parts of times: [25532712. 27833694. 27224996. 29648036.  5128086.   741908. 24729552.
  1925417. 20942870.  8967808. 25146654.  1748891. 13351336. 10693660.
 13416485.  3597684. 17521462. 18918528.  4695088.  6553347.  8299102.
 18166380.  9954801.  6885897. 11551395.  4344278.  8342886. 13112917.
 19087864. 12914089. 22474976.  7784896. 28348196.   637694.  1676195.
 29705352. 10254954. 25429552.  2308074. 23147192. 28037952. 14969731.
 25271148.  4355358. 14672435.   327019. 27210808. 21440628. 20880098.
  9115422.  3808844. 18101924. 14551837. 23926398. 23761846.  3709543.
 18604172.  2410266. 22057268. 15871605.  2363961.  7820516. 12967402.
  9808411. 19773336. 12841123. 20281872.  4744174. 23822910. 22872556.
 16586728. 28209056. 19773058.  1073249.  5400308. 16182668. 28639960.
 16812240.  1738512. 27597636.   274169. 24097612. 28778904. 25754620.
 30051776. 20947376. 27722358. 12179687. 18746916.  8712278.  7125361.
 16832288. 11076337. 10668201. 24004116.  8393550. 10484514. 25165880.
  3232347. 28035596.]

Random parts of labels: [ 29. 305.  11.  52.   5. 135. 140. 145. 117. 233.  97.  52.  77. 127.
 145. 127. 184.  93.  10. 288.  64. 256. 145. 245. 124. 207. 299. 100.
 309. 135. 168. 182. 233. 100. 124. 272. 244. 144.  96. 128. 184. 223.
 121. 223. 159. 269.   2. 277.  79.  83.  11. 281. 277. 140. 231.  60.
  18. 143. 305. 145. 240. 125. 299. 167.  77. 318. 140.  97. 300.  10.
 300.  88. 103.  41. 186.  83. 277. 141. 226. 272.  14. 127. 224. 245.
 116. 194. 299. 184. 309.  53. 224. 318. 168.  88.  50. 113.  52. 224.
 223. 127.]

Shape of X: (3, 109122726)

Done.

Elapsed time for processor ms4alg.sort: 4964.133 sec
[ Saving to process cache ... ]
[ Removing temporary directory ... ]
[ Done. ]

qt-mountainview is taking a long time to process the files now (>15mins already), but I hope it works. If it doesn't, I'll let you know.

[alexmorley]

Glad you got it sorted. I'm going to leave this open but change the title as I think there is still a bug here (if it was just your computer's memory I would expect a problem during a different stage, and/or at least a more obvious error). Out of interest could you say how long your recording was in seconds, and how many channels?

[shashwatsridhar]

This file contained about 729.6 seconds of recording at 30kHz across 96 channels, from a Utah array.