search

SpikeInterface / spikeinterface

A Python-based module for creating flexible and robust spike sorting pipelines.
https://spikeinterface.readthedocs.io
MIT License
488 stars 187 forks source link

dendro demo broken: parallel write #2732

Closed bendichter closed 2 months ago

bendichter commented 4 months ago 
[dendro] Running job: 1b0ca739.e31f3a60
[dendro] Processor: spikeinterface_pipeline_mountainsort5
[dendro] Assembling context
[dendro] Input: input
[dendro] Output: output
[dendro] Parameter: job_kwargs.n_jobs = 0.5
[dendro] Parameter: job_kwargs.chunk_duration = 1s
[dendro] Parameter: job_kwargs.progress_bar = False
[dendro] Parameter: recording_context.electrical_series_path = /acquisition/ElectricalSeriesAp
[dendro] Parameter: recording_context.lazy_read_input = True
[dendro] Parameter: recording_context.write_recording = False
[dendro] Parameter: recording_context.stub_test = True
[dendro] Parameter: recording_context.stub_test_duration_sec = 300
[dendro] Parameter: recording_context.stub_test_num_channels = -1
[dendro] Parameter: run_preprocessing = True
[dendro] Parameter: preprocessing_context.add_preprocessed_to_output_nwb = False
[dendro] Parameter: preprocessing_context.preprocessing_strategy = cmr
[dendro] Parameter: preprocessing_context.highpass_filter.freq_min = 300
[dendro] Parameter: preprocessing_context.highpass_filter.margin_ms = 5
[dendro] Parameter: preprocessing_context.phase_shift.margin_ms = 100
[dendro] Parameter: preprocessing_context.detect_bad_channels.method = coherence+psd
[dendro] Parameter: preprocessing_context.detect_bad_channels.dead_channel_threshold = -0.5
[dendro] Parameter: preprocessing_context.detect_bad_channels.noisy_channel_threshold = 1
[dendro] Parameter: preprocessing_context.detect_bad_channels.outside_channel_threshold = -0.3
[dendro] Parameter: preprocessing_context.detect_bad_channels.n_neighbors = 11
[dendro] Parameter: preprocessing_context.detect_bad_channels.seed = 0
[dendro] Parameter: preprocessing_context.common_reference.reference = global
[dendro] Parameter: preprocessing_context.common_reference.operator = median
[dendro] Parameter: preprocessing_context.highpass_spatial_filter.n_channel_pad = 60
[dendro] Parameter: preprocessing_context.highpass_spatial_filter.n_channel_taper = None
[dendro] Parameter: preprocessing_context.highpass_spatial_filter.direction = y
[dendro] Parameter: preprocessing_context.highpass_spatial_filter.apply_agc = True
[dendro] Parameter: preprocessing_context.highpass_spatial_filter.agc_window_length_s = 0.01
[dendro] Parameter: preprocessing_context.highpass_spatial_filter.highpass_butter_order = 3
[dendro] Parameter: preprocessing_context.highpass_spatial_filter.highpass_butter_wn = 0.01
[dendro] Parameter: preprocessing_context.motion_correction.strategy = skip
[dendro] Parameter: preprocessing_context.motion_correction.preset = nonrigid_accurate
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.detect_kwargs.method = locally_exclusive
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.detect_kwargs.peak_sign = neg
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.detect_kwargs.detect_threshold = 8
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.detect_kwargs.exclude_sweep_ms = 0.1
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.detect_kwargs.radius_um = 50
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.localize_peaks_kwargs.radius_um = 75
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.localize_peaks_kwargs.max_distance_um = 150
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.localize_peaks_kwargs.optimizer = minimize_with_log_penality
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.localize_peaks_kwargs.enforce_decrease = True
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.localize_peaks_kwargs.feature = ptp
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.method = decentralized
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.direction = y
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.bin_duration_s = 2
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.rigid = False
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.bin_um = 5
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.margin_um = 0
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.win_shape = gaussian
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.win_step_um = 100
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.win_sigma_um = 200
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.histogram_depth_smooth_um = 5
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.histogram_time_smooth_s = None
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.pairwise_displacement_method = conv
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.max_displacement_um = 100
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.weight_scale = linear
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.error_sigma = 0.2
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.conv_engine = numpy
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.torch_device = 
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.batch_size = 1
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.corr_threshold = 0
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.time_horizon_s = None
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.convergence_method = lsmr
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.soft_weights = False
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.normalized_xcorr = True
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.centered_xcorr = True
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.temporal_prior = True
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.spatial_prior = False
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.force_spatial_median_continuity = False
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.reference_displacement = median
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.reference_displacement_time_s = 0
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.robust_regression_sigma = 2
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.estimate_motion_kwargs.weight_with_amplitude = False
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.interpolate_motion_kwargs.direction = 1
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.interpolate_motion_kwargs.border_mode = remove_channels
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.interpolate_motion_kwargs.spatial_interpolation_method = idw
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.interpolate_motion_kwargs.sigma_um = 20
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.interpolate_motion_kwargs.p = 1
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_nonrigid_accurate.interpolate_motion_kwargs.num_closest = 3
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.detect_kwargs.method = locally_exclusive
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.detect_kwargs.peak_sign = neg
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.detect_kwargs.detect_threshold = 8
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.detect_kwargs.exclude_sweep_ms = 0.1
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.detect_kwargs.radius_um = 50
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.localize_peaks_kwargs.radius_um = 75
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.localize_peaks_kwargs.feature = ptp
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.method = decentralized
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.direction = y
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.bin_duration_s = 2
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.rigid = False
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.bin_um = 5
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.margin_um = 0
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.win_shape = gaussian
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.win_step_um = 100
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.win_sigma_um = 200
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.histogram_depth_smooth_um = 5
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.histogram_time_smooth_s = None
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.pairwise_displacement_method = conv
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.max_displacement_um = 100
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.weight_scale = linear
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.error_sigma = 0.2
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.conv_engine = numpy
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.torch_device = 
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.batch_size = 1
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.corr_threshold = 0
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.time_horizon_s = None
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.convergence_method = lsmr
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.soft_weights = False
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.normalized_xcorr = True
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.centered_xcorr = True
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.temporal_prior = True
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.spatial_prior = False
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.force_spatial_median_continuity = False
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.reference_displacement = median
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.reference_displacement_time_s = 0
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.robust_regression_sigma = 2
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.estimate_motion_kwargs.weight_with_amplitude = False
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.interpolate_motion_kwargs.direction = 1
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.interpolate_motion_kwargs.border_mode = remove_channels
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.interpolate_motion_kwargs.spatial_interpolation_method = idw
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.interpolate_motion_kwargs.sigma_um = 20
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.interpolate_motion_kwargs.p = 1
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_rigid_fast.interpolate_motion_kwargs.num_closest = 3
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.detect_kwargs.method = locally_exclusive
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.detect_kwargs.peak_sign = neg
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.detect_kwargs.detect_threshold = 8
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.detect_kwargs.exclude_sweep_ms = 0.1
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.detect_kwargs.radius_um = 50
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.localize_peaks_kwargs.radius_um = 40
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.localize_peaks_kwargs.upsampling_um = 5
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.localize_peaks_kwargs.sigma_um = [5, 25, 5]
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.localize_peaks_kwargs.sigma_ms = 0.25
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.localize_peaks_kwargs.margin_um = 30
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.localize_peaks_kwargs.percentile = 10
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.localize_peaks_kwargs.sparsity_threshold = 0.01
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.estimate_motion_kwargs.bin_duration_s = 2
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.estimate_motion_kwargs.rigid = False
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.estimate_motion_kwargs.win_step_um = 50
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.estimate_motion_kwargs.win_sigma_um = 150
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.estimate_motion_kwargs.margin_um = 0
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.estimate_motion_kwargs.win_shape = rect
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.interpolate_motion_kwargs.direction = 1
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.interpolate_motion_kwargs.border_mode = remove_channels
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.interpolate_motion_kwargs.spatial_interpolation_method = idw
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.interpolate_motion_kwargs.sigma_um = 20
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.interpolate_motion_kwargs.p = 1
[dendro] Parameter: preprocessing_context.motion_correction.motion_kwargs_kilosort_like.interpolate_motion_kwargs.num_closest = 3
[dendro] Parameter: preprocessing_context.remove_out_channels = False
[dendro] Parameter: preprocessing_context.remove_bad_channels = False
[dendro] Parameter: preprocessing_context.max_bad_channel_fraction_to_remove = 1.1
[dendro] Parameter: run_spikesorting = True
[dendro] Parameter: sorter_name = mountainsort5
[dendro] Parameter: spikesorting_context.scheme = 2
[dendro] Parameter: spikesorting_context.detect_threshold = 5.5
[dendro] Parameter: spikesorting_context.detect_sign = -1
[dendro] Parameter: spikesorting_context.detect_time_radius_msec = 0.5
[dendro] Parameter: spikesorting_context.snippet_T1 = 20
[dendro] Parameter: spikesorting_context.snippet_T2 = 20
[dendro] Parameter: spikesorting_context.npca_per_channel = 3
[dendro] Parameter: spikesorting_context.npca_per_subdivision = 10
[dendro] Parameter: spikesorting_context.snippet_mask_radius = 250
[dendro] Parameter: spikesorting_context.scheme1_detect_channel_radius = 150
[dendro] Parameter: spikesorting_context.scheme2_phase1_detect_channel_radius = 200
[dendro] Parameter: spikesorting_context.scheme2_detect_channel_radius = 50
[dendro] Parameter: spikesorting_context.scheme2_max_num_snippets_per_training_batch = 200
[dendro] Parameter: spikesorting_context.scheme2_training_duration_sec = 300
[dendro] Parameter: spikesorting_context.scheme2_training_recording_sampling_mode = uniform
[dendro] Parameter: spikesorting_context.scheme3_block_duration_sec = 1800
[dendro] Parameter: spikesorting_context.freq_min = 300
[dendro] Parameter: spikesorting_context.freq_max = 6000
[dendro] Parameter: spikesorting_context.filter = True
[dendro] Parameter: spikesorting_context.whiten = True
[dendro] Parameter: run_postprocessing = True
[dendro] Parameter: postprocessing_context.sparsity.method = radius
[dendro] Parameter: postprocessing_context.sparsity.radius_um = 100
[dendro] Parameter: postprocessing_context.waveforms_raw.ms_before = 1
[dendro] Parameter: postprocessing_context.waveforms_raw.ms_after = 2
[dendro] Parameter: postprocessing_context.waveforms_raw.max_spikes_per_unit = 100
[dendro] Parameter: postprocessing_context.waveforms_raw.return_scaled = True
[dendro] Parameter: postprocessing_context.waveforms_raw.use_relative_path = True
[dendro] Parameter: postprocessing_context.waveforms.ms_before = 3
[dendro] Parameter: postprocessing_context.waveforms.ms_after = 4
[dendro] Parameter: postprocessing_context.waveforms.max_spikes_per_unit = 500
[dendro] Parameter: postprocessing_context.waveforms.return_scaled = True
[dendro] Parameter: postprocessing_context.waveforms.use_relative_path = True
[dendro] Parameter: postprocessing_context.spike_amplitudes.peak_sign = neg
[dendro] Parameter: postprocessing_context.spike_amplitudes.return_scaled = True
[dendro] Parameter: postprocessing_context.spike_amplitudes.outputs = concatenated
[dendro] Parameter: postprocessing_context.similarity.method = cosine_similarity
[dendro] Parameter: postprocessing_context.correlograms.window_ms = 100
[dendro] Parameter: postprocessing_context.correlograms.bin_ms = 2
[dendro] Parameter: postprocessing_context.isis.window_ms = 100
[dendro] Parameter: postprocessing_context.isis.bin_ms = 5
[dendro] Parameter: postprocessing_context.locations.method = monopolar_triangulation
[dendro] Parameter: postprocessing_context.template_metrics.upsampling_factor = 10
[dendro] Parameter: postprocessing_context.principal_components.n_components = 5
[dendro] Parameter: postprocessing_context.principal_components.mode = by_channel_local
[dendro] Parameter: postprocessing_context.principal_components.whiten = True
[dendro] Parameter: postprocessing_context.quality_metrics.presence_ratio = True
[dendro] Parameter: postprocessing_context.quality_metrics.snr = True
[dendro] Parameter: postprocessing_context.quality_metrics.isi_violation = True
[dendro] Parameter: postprocessing_context.quality_metrics.rp_violation = True
[dendro] Parameter: postprocessing_context.quality_metrics.sliding_rp_violation = True
[dendro] Parameter: postprocessing_context.quality_metrics.amplitude_cutoff = True
[dendro] Parameter: postprocessing_context.quality_metrics.amplitude_median = True
[dendro] Parameter: postprocessing_context.quality_metrics.nearest_neighbor = False
[dendro] Parameter: postprocessing_context.quality_metrics.nn_isolation = False
[dendro] Parameter: postprocessing_context.quality_metrics.nn_noise_overlap = False
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.presence_ratio.bin_duration_s = 60
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.snr.peak_sign = neg
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.snr.peak_mode = extremum
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.isi_violation.isi_threshold_ms = 1.5
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.isi_violation.min_isi_ms = 0
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.rp_violation.refractory_period_ms = 1
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.rp_violation.censored_period_ms = 0
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.sliding_rp_violation.bin_size_ms = 0.25
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.sliding_rp_violation.window_size_s = 1
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.sliding_rp_violation.exclude_ref_period_below_ms = 0.5
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.sliding_rp_violation.max_ref_period_ms = 10
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.amplitude_cutoff.peak_sign = neg
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.amplitude_cutoff.num_histogram_bins = 100
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.amplitude_cutoff.histogram_smoothing_value = 3
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.amplitude_cutoff.amplitudes_bins_min_ratio = 5
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.amplitude_median.peak_sign = neg
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.nearest_neighbor.max_spikes = 10000
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.nearest_neighbor.min_spikes = 10
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.nearest_neighbor.n_neighbors = 4
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.nn_isolation.max_spikes = 10000
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.nn_isolation.min_spikes = 10
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.nn_isolation.n_neighbors = 4
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.nn_isolation.n_components = 10
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.nn_isolation.radius_um = 100
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.nn_noise_overlap.max_spikes = 10000
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.nn_noise_overlap.min_spikes = 10
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.nn_noise_overlap.n_neighbors = 4
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.nn_noise_overlap.n_components = 10
[dendro] Parameter: postprocessing_context.quality_metrics.qm_params.nn_noise_overlap.radius_um = 100
[dendro] Parameter: postprocessing_context.quality_metrics.n_jobs = 1
[dendro] Parameter: postprocessing_context.duplicate_threshold = 0.9
[dendro] Parameter: run_curation = True
[dendro] Parameter: curation_context.curation_query = isi_violations_ratio < 0.5 and amplitude_cutoff < 0.1 and presence_ratio > 0.8
[dendro] Parameter: run_visualization = True
[dendro] Parameter: visualization_context.recording.timeseries.n_snippets_per_segment = 2
[dendro] Parameter: visualization_context.recording.timeseries.snippet_duration_s = 0.5
[dendro] Parameter: visualization_context.recording.timeseries.skip = False
[dendro] Parameter: visualization_context.recording.drift.detection.peak_sign = neg
[dendro] Parameter: visualization_context.recording.drift.detection.detect_threshold = 5.0
[dendro] Parameter: visualization_context.recording.drift.detection.exclude_sweep_ms = 0.1
[dendro] Parameter: visualization_context.recording.drift.localization.ms_before = 0.1
[dendro] Parameter: visualization_context.recording.drift.localization.ms_after = 0.3
[dendro] Parameter: visualization_context.recording.drift.localization.radius_um = 100.0
[dendro] Parameter: visualization_context.recording.drift.decimation_factor = 30
[dendro] Parameter: visualization_context.recording.drift.alpha = 0.15
[dendro] Parameter: visualization_context.recording.drift.vmin = -200
[dendro] Parameter: visualization_context.recording.drift.vmax = 0
[dendro] Parameter: visualization_context.recording.drift.cmap = Greys_r
[dendro] Parameter: visualization_context.recording.label = Recording visualization from SI pipelines
[dendro] Parameter: visualization_context.sorting_summary.unit_table_properties = default_qc
[dendro] Parameter: visualization_context.sorting_summary.curation = True
[dendro] Parameter: visualization_context.sorting_summary.label_choices = SUA, MUA, noise
[dendro] Parameter: visualization_context.sorting_summary.label = Sorting summary from SI pipelines
[dendro] Preparing to run processor
[dendro] Running processor
INFO:common.processor_pipeline:Opening remote input file
INFO:common.processor_pipeline:Creating input recording
INFO:common.processor_pipeline:Running in stub test mode
INFO:common.processor_pipeline:FrameSliceRecording: 384 channels - 30.0kHz - 1 segments - 9,000,000 samples 
                     300.00s (5.00 minutes) - int16 dtype - 6.44 GiB
INFO:common.processor_pipeline:Running pipeline
INFO:spikeinterface_pipelines.logger:Preprocessing recording
INFO:spikeinterface_pipelines.logger:[Preprocessing]    Running Preprocessing stage
INFO:spikeinterface_pipelines.logger:[Preprocessing]    Duration: 300.0 s
INFO:spikeinterface_pipelines.logger:[Preprocessing]    Phase shift
INFO:spikeinterface_pipelines.logger:[Preprocessing]    Bad channel detection found: 10 dead, 0 noise, 0 out channels
INFO:spikeinterface_pipelines.logger:[Spikesorting]     Starting mountainsort5 spike sorter
Warning! The recording is already filtered, but mountainsort5 filter is enabled
filtering
whitening
Error running mountainsort5
INFO:spikeinterface_pipelines.logger:Spike sorting error:
Spike sorting error trace:
Traceback (most recent call last):
  File "/src/spikeinterface/src/spikeinterface/sorters/basesorter.py", line 258, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File "/src/spikeinterface/src/spikeinterface/sorters/external/mountainsort5.py", line 191, in _run_from_folder
    recording_cached = create_cached_recording(
  File "/usr/local/lib/python3.10/site-packages/mountainsort5/util/create_cached_recording.py", line 18, in create_cached_recording
    si.BinaryRecordingExtractor.write_recording(
  File "/src/spikeinterface/src/spikeinterface/core/binaryrecordingextractor.py", line 148, in write_recording
    write_binary_recording(recording, file_paths=file_paths, dtype=dtype, **job_kwargs)
  File "/src/spikeinterface/src/spikeinterface/core/recording_tools.py", line 137, in write_binary_recording
    executor.run()
  File "/src/spikeinterface/src/spikeinterface/core/job_tools.py", line 401, in run
    for res in results:
  File "/usr/local/lib/python3.10/concurrent/futures/process.py", line 575, in _chain_from_iterable_of_lists
    for element in iterable:
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Spike sorting failed. You can inspect the runtime trace in /tmp/working/scratch/tmp_spikesorting/spikeinterface_log.json.
Traceback (most recent call last):
  File "/app/main.py", line 44, in <module>
    app.run()
  File "/src/dendro/python/dendro/sdk/App.py", line 84, in run
    return _run_job_child_process(job_id=JOB_ID, job_private_key=JOB_PRIVATE_KEY, processors=self._processors)
  File "/src/dendro/python/dendro/sdk/_run_job_child_process.py", line 141, in _run_job_child_process
    processor_class.run(context)
  File "/app/main.py", line 37, in run
    run_pipeline(context)
  File "/app/common/processor_pipeline.py", line 116, in run_pipeline
    recording_preprocessed, sorting, waveform_extractor, sorting_curated, visualization_output = si_pipeline.run_pipeline(
  File "/src/spikeinterface_pipelines/src/spikeinterface_pipelines/pipeline.py", line 114, in run_pipeline
    raise Exception("Spike sorting failed")
Exception: Spike sorting failed

This occurred during @luiztauffer 's Dendro tutorial

alejoe91 commented 4 months ago

Thanks @bendichter

This is a known issue and we pushed a patch on the MS5 wrapper to fix it. I guess @luiztauffer still needs to update the app image to include that fix

bendichter commented 4 months ago

oh, good to know. Thanks!

zm711 commented 2 months ago

I close this for now, but let us know if it is still a problem.