Training Network

[BavoKempen]

Dear,

The Detection networks for mice do not work very well on a certain amount of data that I have been asked to analyze (detect/classify) due to very noisy recordings (both low and high frequency). Consequently, I would like to try and train a network to be less sensitive to this noise or with a lower signal to noise ratio. However, I am a bit stuck given that I only have the .wav files. Furthermore, I explored the option to import from ethovision (since I already manually detected 30 recordings), however, without success. Therefore, it would be really handy to be able to also go through the spectrogram manually and indicate USV's manually and use this as input for training a network. If by any chance I missed that option given my very limited matlab and ANN knowledge I sincerely apologize beforehand. Also, I ran the pre-trained networks (mice) on somewhat cleaner data and it worked very well!

In sum, it is not entirely clear how to train a network from the manual and which files are expected and why and in which way we are able to obtain these (for a novice).

Thank you in advance and as well for making DeepSqueak!

Kind regards,

Bavo

[MxMarx]

I just found it fixed a bug in the code for importing from ultravox, so hopefully it works now. We mostly use Raven Lite for manually detecting recordings. If it still isn't working, you could send us the file and we'll take a look at it.

If you are getting a lot of false positives, you could also try running the files through "Tools>Automatic Review>Post Hoc Denoising"

Good luck!

[BavoKempen]

Example data.zip

Dear MxMarx, Firstly, I would like to thank you for your fast response. Also, I corrected both scripts that were updated concerning merging and UltraVox XT. Nonetheless, when I try to import the call statistics I get these errors from the UltraVox XT import function:

Error using fopen
File identifier must be an integer-valued scalar of type double.

Error in Import_From_Ultravox_Callback (line 10)
fin = fopen([ultravoxPath ultravoxName],'r');

Error in gui_mainfcn (line 95)
        feval(varargin{:});

Error in DeepSqueak (line 30)
    gui_mainfcn(gui_State, varargin{:});

Error in matlab.graphics.internal.figfile.FigFile/read>@(hObject,eventdata)DeepSqueak('Import_From_Ultravox_Callback',hObject,eventdata,guidata(hObject))

Error while evaluating Menu Callback.

Warning: Variable names were modified to make them valid MATLAB identifiers. The original names are saved in the VariableDescriptions property. 
Undefined operator '-' for input arguments of type 'cell'.

Error in Import_From_Ultravox_Callback (line 39)
        ultravox.StopTime_s_(i) - ultravox.StartTime_s_(i),...

Error in gui_mainfcn (line 95)
        feval(varargin{:});

Error in DeepSqueak (line 30)
    gui_mainfcn(gui_State, varargin{:});

Error in matlab.graphics.internal.figfile.FigFile/read>@(hObject,eventdata)DeepSqueak('Import_From_Ultravox_Callback',hObject,eventdata,guidata(hObject))

Error while evaluating Menu Callback.

Therefore, I suspect that I am trying to import the wrong 'log' file, namely, the 'call statistics' exported from the computer containing the Noldus software. However, given that in the function in Matlab is indicated that it should be a file with the '/*.txt' extension this seems the only file reasonable containing USV data in that format. Attached you will find one recording in .wav format and the aforementioned .txt file within the .zip file "example data" (given that GitHub does not seem to support .wav).

Thank you in advance!

Kind regards,

Bavo Kempen

[MxMarx]

I tried using your UltraVox file, and the same thing happen to me. I took a look at the file, and I think I found the problem. My version of UltraVox exports files with periods as the decimal separator, but your version appears to use commas. I added a line that replaces commas with periods, and it worked for me after I did that.

Looking at the audio file, it seems that the broadband noise stops around 40 kHz, so setting that as the low frequency cutoff might help. The file is also rather quiet, but I was able to pick up a few calls after setting the spectrogram gain to 5.

[BavoKempen]

Dear MxMarx,

That seemed indeed to be the problem, I can now read in the data from UltraVox. Very grateful for the quick response and help! Also, do you think that the network has the capacity to after some retraining be able to detect data that is rather noisy? I will try it anyways, it would be great if it works. Tips are always welcome!

Again thanks a lot for the quick and good help. Kind regards,

Bavo

[BavoKempen]

Dear,

After preparing everything to train an existing network I bounced upon the error underneath which occurs during the first step of training.

Training a Faster R-CNN Object Detector for the following object classes:

* USV

Step 1 of 4: Training a Region Proposal Network (RPN).
Training on single CPU.
|========================================================================================|
|  Epoch  |  Iteration  |  Time Elapsed  |  Mini-batch  |  Mini-batch  |  Base Learning  |
|         |             |   (hh:mm:ss)   |   Accuracy   |     RMSE     |      Rate       |
|========================================================================================|
|       1 |           1 |       00:00:00 |       87.50% |         1.55 |      1.0000e-05 |
|       1 |          50 |       00:00:07 |       39.84% |         0.71 |      1.0000e-05 |

Error using builtin
Padding exceeds array bounds.

Error in nnet.internal.cnnhost.convolveBackwardFilter2D (line 31)
dLossdW = builtin('_batchconvBackward', X, dLossdZ, ...

Error in nnet.internal.cnn.layer.util.Convolution2DHostStrategy/backward (line 60)
                    dW{1} = nnet.internal.cnnhost.convolveBackwardFilter2D( ...

Error in nnet.internal.cnn.layer.Convolution2D/doBackward (line 385)
            [varargout{1:nargout}] = this.ExecutionStrategy.backward( ...

Error in nnet.internal.cnn.layer.Convolution2D/backwardNormal (line 359)
            [varargout{1:nargout}] = this.doBackward(X, this.Weights.Value, dZ);

Error in nnet.internal.cnn.layer.Convolution2D/backward (line 152)
                [varargout{1:nargout}] = this.backwardNormal(X, [], dZ, []);

Error in vision.internal.cnn.internalRPNSeriesNetwork/computeGradientsForTraining/efficientBackProp (line 400)
                    [dX, gradientsThisLayer] = this.Layers{currentLayer}.backward( backwardArgs{:} );

Error in vision.internal.cnn.internalRPNSeriesNetwork>@()efficientBackProp(el) (line 476)
                    @()efficientBackProp(el),...

Error in vision.internal.cnn.internalRPNSeriesNetwork>iExecuteWithStagedGPUOOMRecovery (line 757)
        [ varargout{1:nOutputs} ] = computeFun();

Error in vision.internal.cnn.internalRPNSeriesNetwork/computeGradientsForTraining (line 475)
                thisDw =  iExecuteWithStagedGPUOOMRecovery( ...

Error in vision.internal.cnn.FastRCNNTrainer/computeGradients (line 162)
            [gradients, predictions, states] = net.computeGradientsForTraining(X, Y, needsStatefulTraining,
            propagateState);

Error in vision.internal.cnn.FastRCNNTrainer/train (line 77)
                    [gradients, predictions, ~] = this.computeGradients(net, X, response, needsStatefulTraining,
                    propagateState);

Error in vision.internal.cnn.trainNetwork (line 52)
internalNetwork = trainer.train(internalNetwork, dispatcher);

Error in fasterRCNNObjectDetector.trainRPN (line 175)
            [rpn, info] = vision.internal.cnn.trainNetwork(dispatcher, layers, opts, checkpointSaver);

Error in trainFasterRCNNObjectDetector (line 289)
    [stage1Detector, rpn, info] = fasterRCNNObjectDetector.trainRPN(trainingData, rpn, options(1),
    iStageOneParams(params), checkpointSaver);

Error in TrainSqueakDetector (line 48)
    detector = trainFasterRCNNObjectDetector(TrainingTables, layers, options, ...

Error in trainnew_Callback (line 32)
        [detector layers, options] = TrainSqueakDetector(TrainingTables,layers);

Error in gui_mainfcn (line 95)
        feval(varargin{:});

Error in DeepSqueak (line 30)
    gui_mainfcn(gui_State, varargin{:});

Error in
matlab.graphics.internal.figfile.FigFile/read>@(hObject,eventdata)DeepSqueak('trainnew_Callback',hObject,eventdata,guidata(hObject)) 
Error while evaluating Menu Callback.

C

Any idea why this might occur? Also, I cannot attach the .mat file here it seems, nonetheless, I will attach the corresponding image map of "T1.mat" . Underneath two screenshots of the data's path.

T1.zip

Thank you in advance and sorry for bothering you again (if you need anything else, I will be happy to provide it).

Kind regards,

Bavo

[DrCoffey]

Hey BavoKempen,

Sorry for taking so long to get back to you. Can you email (mrcoffey[at]uw.edu) me the .mat files with the training tables?

The training started and the images look fine, so it may be a matlab version issue. What version are you using?

Thanks, Kevin

[BavoKempen]

Dear Kevin,

I emailed you the files producing the respective errors. Also, Matlab R2018a.

Kind regards,

Bavo

[DrCoffey]

Hey Bavo,

We updated the network training code and I am training a network with your data as we speak. I will send you the network as soon as it is done. If the network is functional, do you mind if I upload it to the main branch?

Thanks, Kevin

[BavoKempen]

Dear Kevin,

That is definitely not a problem and the goal that we are trying to achieve. Nonetheless, over the past weeks I have been gathering pre-detected data in Ultravox concerning the same experiments and thus with the same noise and so on (currently 18k USVs). Consequently, we have arranged to train a network from scratch on the "supercomputer" with 90% of this data and keep 10% as a validation. Thus, I will keep you posted of the results and if it works, send it to you as well to upload it to the main branch. If you have any tips concerning certain values that work the best that would be great. For instance, when reading in the data from Ultravox to create the .mat file I have to specify the call bandwidth. Should I just specify the bandwidth then of the USV with the largest frequency range in the concerning file, e.g 47.5kHz, or rather go over this and choose a value like 60kHz. Furthermore, given our large sample of USV's, should we create 2 replications with noise when creating the training tables and images. Finally, if you have any other recommendations for the other variables (e.g. FFT specifics) when creating training tables and images they would be more than welcome!

Kind regards,

Bavo Kempen

[DrCoffey]

Hey Bavo,

While working on your issue yesterday we discovered that newly trained networks are behaving different then the original set we trained. We think Matlab changed how faster-RCNN handles the intensity of the training images (0-1) vs (0-255). We are going to try and figure it out early next week and update the training code. Until then I would hold off on training that massive network! I'll let you know asap when it is fixed and I'll answer all your training questions at the same time!

Thanks for your patience, Kevin

On Sat, Dec 8, 2018, 5:14 AM BavoKempen <notifications@github.com wrote:

Dear Kevin,

That is definitely not a problem and the goal that we are trying to achieve. Nonetheless, over the past weeks I have been gathering pre-detected data in Ultravox concerning the same experiments and thus with the same noise and so on (currently 18k USVs). Consequently, we have arranged to train a network from scratch on the "supercomputer" with 90% of this data and keep 10% as a validation. Thus, I will keep you posted of the results and if it works, send it to you as well to upload it to the main branch. If you have any tips concerning certain values that work the best that would be great. For instance, when reading in the data from Ultravox to create the .mat file I have to specify the call bandwith. Should I just specify the bandwidth then of the USV with the largest frequency range in the concerning file, e.g 47.5kHz, or rather go over this and choose a value like 60kHz. Furthermore, given our large sample of USV's, should we create 2 replications with noise when creating the training tables and images. Finally, if you have any other recommendations for the other variables (e.g. FFT specifics) when creating training tables and images they would be more than welcome!

Kind regards,

Bavo Kempen

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/DrCoffey/DeepSqueak/issues/7#issuecomment-445458297, or mute the thread https://github.com/notifications/unsubscribe-auth/AlxTE6GHre3hoH_kxf3DMIBN1QkYVi-vks5u27r7gaJpZM4Yo3zo .

[BavoKempen]

Dear Kevin,

Sounds good, I will hold off until I hear back from you. Also, thank you for the upcoming training advice, much appreciated!

Furthermore, I also encountered an error when using solely the LongRatCall_Network.mat for detection purposes on the same data as the previous examples came from. Consequently, I have been looking a bit at the same errors on fora concering the CNN package, however, I found no answers there. It is of not too much importance for me at the moment, nonetheless, I wanted to let you know.

Index exceeds array bounds.

Error in CalculateStats (line 45)
stats.SignalToNoise = mean(1 - stats.Entropy(stats.ridgeTime));

Error in SqueakDetect (line 238)
            stats = CalculateStats(I,wind,noverlap,nfft,Calls(i).Rate,Calls(i).Box,EntropyThreshold,AmplitudeThreshold,0);

Error in multinetdect_Callback (line 93)
            Calls1=SqueakDetect(AudioFile,NeuralNetwork,handles.audiofiles(CurrentAudioFile).name,Settings(:,k),0,0,j,length(audioselections),networkname);

Error in gui_mainfcn (line 95)
        feval(varargin{:});

Error in DeepSqueak (line 30)
    gui_mainfcn(gui_State, varargin{:});

Error in
matlab.graphics.internal.figfile.FigFile/read>@(hObject,eventdata)DeepSqueak('multinetdect_Callback',hObject,eventdata,guidata(hObject),0)

Error while evaluating UIControl Callback.

Kind regards and looking forward to hearing from you soon,

Bavo Kempen

[DrCoffey]

Hey Bavo,

We fixed the error when using the LongRatCall_Network! I'll update you on the network training ASAP.

Kevin

[DrCoffey]

Hey, sorry we only fixed it in the 1.2 branch not the master branch. You can download and try that, or wait until we merge them over.

[BavoKempen]

Dear Kevin,

That's okay, it is not of vital importance yet. Looking forward to the network training information! Kind regards,

Bavo Kempen

[DrCoffey]

Hey Bavo,

DeepSqueak 2.0 has been uploaded and it makes major improvements to everything, including the network training. I would try re-detecting your files with our new networks before going ahead with training your own, they seem to be a lot more sensitive and deal with a huge range of volumes.

If you do choose to train your own network. Faster Rcnn training doesn't work correctly in 2018b, so be sure to use 2018a.

If you don't have the bandwidth of each call you will have to box using the maximum and minimum. It would be ideal to have the calls cleanly boxed, but there is no reason you can't use oversize boxes. The resulting network will detect with larger boxes though.

The replicates may not be necessary, but we usually find it helps with generalization.

Also, we train on bouts of calls (~1s) because it leaves more noise in the background. If you have the original audio files, I recommend creating training images in bouts. (its an option in the "create training images" menu.

The spectrogram settings will determine the quality of the images and the speed of detection. Our default settings in 2.0 are a good middle ground.

Hope that helps! let me know if you have any problems with DeepSqueak 2.0!

[BavoKempen]

Dear Kevin,

My apologies for taking this long to get back to you. Indeed, the networks in DeepSqueak 2.0 seem more sensitive than the ones in previous versions, however, not yet satisfactory. Therefore, we will go ahead and train our own network anyways to see if certain lab specific background noise etc might play a role. Furthermore, I will follow your recommendations and train on bouts instead of individual calls since I have the original audiofiles. Also, peculiar enough the MouseCallNetwork_V2 is heavily underperforming compared to the ShortRatCallNetwork_V2 and AllShortCallNetwork_V1. The reason why this seems weird in a sense is that the USV data that we have is from mice. Consequently, I was thinking if there might be strain specific properties in the calls similar to gender differences in the calls. Therefore, does the training data of the MouseCallNetwork_V1/V2 originate from C57/Bl6 mice or another strain and additionally, do they come from the same gender?

Also, the multi detect function when using two networks seems to still have some buggs when merging. I think these derive from the default that when a network does not detect an USV it fails to save and hence fails to merge the two .mat files from the two respective networks or something along these lines. e.g.

Undefined function or variable 'audio'.

Error in Automerge_Callback (line 62)
        audio=[pad; audio];

Error in multinetdect_Callback (line 124)
            Calls = Automerge_Callback(Calls1,Calls2,AudioFile);

Error in gui_mainfcn (line 95)
        feval(varargin{:});

Error in DeepSqueak (line 30)
    gui_mainfcn(gui_State, varargin{:});

Error in
matlab.graphics.internal.figfile.FigFile/read>@(hObject,eventdata)DeepSqueak('multinetdect_Callback',hObject,eventdata,guidata(hObject),0) 
Error while evaluating UIControl Callback.

Kind regards and thanks again for all the help,

Bavo Kempen

[DrCoffey]

Hey Bavo,

The networks were trained on male mice vocalizations from a single strain. We are currently working on a multi-strain network.

Also I fixed the merge bug!

-Kevin

[BavoKempen]

Dear Kevin,

That's great, it might indeed be due to this then since we both have male and female data (only from one strain though). Also, fixing that merge bug concurrently fixed an issue with ultravox import, so thanks for that!

Kind regards,

Bavo Kempen

[DrCoffey]

No problem! I'm going to close this issue. Feel free to open another one if you find a new bug!