JaneliaSciComp / JRCLUST

JRCLUST
Apache License 2.0
34 stars 22 forks source link

bug- spitting in feature projection is splitting in time #128

Open WeissShahaf opened 5 years ago

WeissShahaf commented 5 years ago

when splitting clusters using feature projection, the split always seems to be a time cut off in addition to that, the clusters which appear to be distinct, are still present after cutting.

this happens on various computers, and recording sessions.

GL383_20190815_OF_g0_t0.imec0.ap.prm.zip

If you have a usage or support question, come join our Gitter and ask your question there. If you found a bug, please provide all of the following:

Output of ver

ver

MATLAB Version: 9.6.0.1072779 (R2019a) MATLAB License Number: 40523909 Operating System: Microsoft Windows 10 Enterprise Version 10.0 (Build 18362) Java Version: Java 1.8.0_181-b13 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode

MATLAB Version 9.6 (R2019a) Simulink Version 9.3 (R2019a) 5G Toolbox Version 1.1 (R2019a) AUTOSAR Blockset Version 2.0 (R2019a) Aerospace Blockset Version 4.1 (R2019a) Aerospace Toolbox Version 3.1 (R2019a) Antenna Toolbox Version 4.0 (R2019a) Audio Toolbox Version 2.0 (R2019a) Automated Driving Toolbox Version 2.0 (R2019a) Bioinformatics Toolbox Version 4.12 (R2019a) Communications Toolbox Version 7.1 (R2019a) Computer Vision Toolbox Version 9.0 (R2019a) Control System Toolbox Version 10.6 (R2019a) Curve Fitting Toolbox Version 3.5.9 (R2019a) DSP System Toolbox Version 9.8 (R2019a) Data Acquisition Toolbox Version 4.0 (R2019a) Database Toolbox Version 9.1 (R2019a) Datafeed Toolbox Version 5.8.1 (R2019a) Deep Learning Toolbox Version 12.1 (R2019a) Econometrics Toolbox Version 5.2 (R2019a) Embedded Coder Version 7.2 (R2019a) Filter Design HDL Coder Version 3.1.5 (R2019a) Financial Instruments Toolbox Version 2.9 (R2019a) Financial Toolbox Version 5.13 (R2019a) Fixed-Point Designer Version 6.3 (R2019a) Fuzzy Logic Toolbox Version 2.5 (R2019a) GPU Coder Version 1.3 (R2019a) GUI Layout Toolbox Version 2.3.4 (R2018b) Global Optimization Toolbox Version 4.1 (R2019a) HDL Coder Version 3.14 (R2019a) HDL Verifier Version 5.6 (R2019a) Image Acquisition Toolbox Version 6.0 (R2019a) Image Processing Toolbox Version 10.4 (R2019a) Instrument Control Toolbox Version 4.0 (R2019a) LTE HDL Toolbox Version 1.3 (R2019a) LTE Toolbox Version 3.1 (R2019a) MATLAB Coder Version 4.2 (R2019a) MATLAB Compiler Version 7.0.1 (R2019a) MATLAB Compiler SDK Version 6.6.1 (R2019a) MATLAB Report Generator Version 5.6 (R2019a) Mapping Toolbox Version 4.8 (R2019a) Mixed-Signal Blockset Version 1.0 (R2019a) Model Predictive Control Toolbox Version 6.3 (R2019a) Model-Based Calibration Toolbox Version 5.6 (R2019a) OPC Toolbox Version 4.0.7 (R2019a) Optimization Toolbox Version 8.3 (R2019a) Parallel Computing Toolbox Version 7.0 (R2019a) Partial Differential Equation Toolbox Version 3.2 (R2019a) Phased Array System Toolbox Version 4.1 (R2019a) Powertrain Blockset Version 1.5 (R2019a) Predictive Maintenance Toolbox Version 2.0 (R2019a) RF Blockset Version 7.2 (R2019a) RF Toolbox Version 3.6 (R2019a) Reinforcement Learning Toolbox Version 1.0 (R2019a) Risk Management Toolbox Version 1.5 (R2019a) Robotics System Toolbox Version 2.2 (R2019a) Robust Control Toolbox Version 6.6 (R2019a) Sensor Fusion and Tracking Toolbox Version 1.1 (R2019a) SerDes Toolbox Version 1.0 (R2019a) Signal Processing Toolbox Version 8.2 (R2019a) SimBiology Version 5.8.2 (R2019a) SimEvents Version 5.6 (R2019a) Simscape Version 4.6 (R2019a) Simscape Driveline Version 2.16 (R2019a) Simscape Electrical Version 7.1 (R2019a) Simscape Fluids Version 2.6 (R2019a) Simscape Multibody Version 6.1 (R2019a) Simulink 3D Animation Version 8.2 (R2019a) Simulink Check Version 4.3 (R2019a) Simulink Code Inspector Version 3.4 (R2019a) Simulink Coder Version 9.1 (R2019a) Simulink Control Design Version 5.3 (R2019a) Simulink Coverage Version 4.3 (R2019a) Simulink Design Optimization Version 3.6 (R2019a) Simulink Design Verifier Version 4.1 (R2019a) Simulink Desktop Real-Time Version 5.8 (R2019a) Simulink PLC Coder Version 3.0 (R2019a) Simulink Real-Time Version 6.10 (R2019a) Simulink Report Generator Version 5.6 (R2019a) Simulink Requirements Version 1.3 (R2019a) Simulink Test Version 3.0 (R2019a) SoC Blockset Version 1.0 (R2019a) Spreadsheet Link Version 3.4.1 (R2019a) Stateflow Version 10.0 (R2019a) Statistics and Machine Learning Toolbox Version 11.5 (R2019a) Symbolic Math Toolbox Version 8.3 (R2019a) System Composer Version 1.0 (R2019a) System Identification Toolbox Version 9.10 (R2019a) Text Analytics Toolbox Version 1.3 (R2019a) Trading Toolbox Version 3.5.1 (R2019a) Vehicle Dynamics Blockset Version 1.2 (R2019a) Vehicle Network Toolbox Version 4.2 (R2019a) Vision HDL Toolbox Version 1.8 (R2019a) WLAN Toolbox Version 2.1 (R2019a) Wavelet Toolbox Version 5.2 (R2019a)

Output of gpuDevice

gpuDevice(1) ans = CUDADevice with properties:

                  Name: 'Quadro RTX 8000'
                 Index: 1
     ComputeCapability: '7.5'
        SupportsDouble: 1
         DriverVersion: 10.1000
        ToolkitVersion: 10
    MaxThreadsPerBlock: 1024
      MaxShmemPerBlock: 49152
    MaxThreadBlockSize: [1024 1024 64]
           MaxGridSize: [2.1475e+09 65535 65535]
             SIMDWidth: 32
           TotalMemory: 4.8318e+10
       AvailableMemory: 4.0199e+10
   MultiprocessorCount: 72
          ClockRateKHz: 1770000
           ComputeMode: 'Default'
  GPUOverlapsTransfers: 1
KernelExecutionTimeout: 0
      CanMapHostMemory: 1
       DeviceSupported: 1
        DeviceSelected: 1

The command you used

(e.g., jrc detect-sort myparam.prm) jrc manual

If manually sorting, what it is you tried to do

split using Feature projection

Console output

2019-08-26 09:00:17 Computing cluster mean waveforms... 2019-08-26 09:00:17 Finished computing cluster mean waveforms (took 0.13 s) 2019-08-26 09:00:17 Computing waveform correlation... 2019-08-26 09:00:17 Finished computing waveform correlation (took 0.12 s) 2019-08-26 09:00:17 Computing cluster quality scores... 2019-08-26 09:00:17 Finished computing cluster quality scores (took 0.02 s)

aliddell commented 5 years ago

Hi @WeissShahaf, can you let me know what's the git commit hash of the version you're running, and maybe show me a screenshot?

WeissShahaf commented 5 years ago

i'm running the latest commit, SHA 5e8f020e4a1adacf8eccaa258836c4c86de702fd

before before1 before2 before3 before4 before5 after1 after2 after3 after3a after5

aliddell commented 5 years ago

Makes sense now. This could be a bug in how the features for the feature projection view are sampled. Let me investigate and get back to you.

aliddell commented 5 years ago

Hey, please give the latest commit a pull and let me know if you still see this problem.,

WeissShahaf commented 5 years ago

-some weird behavior now: projection plot now is displaying spikes for some clusters but not others. though it shows background spikes for all:

1 2

-when i try splitting it fails:

Warning: Failed to split: Operands to the || and && operators must be convertible to logical scalar values.

In jrclust.curate.CurateController/splitCluster (line 16) In jrclust.curate.CurateController/splitPoly (line 68) In jrclust.curate.CurateController/keyPressFigProj (line 65) In jrclust.curate.CurateController/plotAllFigures>@(varargin)obj.keyPressFigProj(varargin{:}) (line 23)

-Also display wave waveforms is weird- mean waveforms seem flat, when waveforms are clearly not:

U235_waveforms u2_mean u2_waveforms U235_mean

and when i try to split it fails

aliddell commented 5 years ago

Can you pull the latest and run the following script for me?

hCfg = jrclust.Config('/path/to/your/params.prm'); % load your prm file
hCfg.autoRecover = 1; % set an optional parameter to try automatically fixing this
hJRC = JRC(hCfg);
hJRC.loadFiles(); % load bin files, res.mat, try to fix any data inconsistencies
hJRC.curate(); % inspect the results
WeissShahaf commented 5 years ago

it does split after the patch, sometimes, but still splitting in time. other cluster still fail to split.

console output: Warning: Failed to split: split indices exceed bounds

In jrclust.curate.CurateController/splitCluster (line 21) In jrclust.curate.CurateController/splitPoly (line 95) In jrclust.curate.CurateController/keyPressFigProj (line 65) In jrclust.curate.CurateController/plotAllFigures>@(varargin)obj.keyPressFigProj(varargin{:}) (line 23)

for some recordings the patch fails. especially if i try it after starting manual curation. then i get an error message about unstable statistics and that curation will likely not be possible.

aliddell commented 5 years ago

That surprises me. I'm inclined to think it's something particular to your dataset. May I take a look at it?

Also, you mentioned it only sometimes splits? What happens when it doesn't?

WeissShahaf commented 5 years ago

if the split fails it says operation failed.

Warning: Failed to split: split indices exceed bounds

In jrclust.curate.CurateController/splitCluster (line 21) In jrclust.curate.CurateController/splitPoly (line 95) In jrclust.curate.CurateController/keyPressFigProj (line 65) In jrclust.curate.CurateController/plotAllFigures>@(varargin)obj.keyPressFigProj(varargin{:}) (line 23)

here is a link to an example dataset test folder: https://www.dropbox.com/sh/36jcwu2n5qkkz5l/AADFZmVmHGaI40jBOXyUOHDWa?dl=0