search

paulbrodersen / somnotate

Automated polysomnography for experimental animal research
GNU General Public License v3.0
14 stars 7 forks source link

low accuracy for test state annotation #18

Open rjp109 opened 2 weeks ago

rjp109 commented 2 weeks ago

Hello,

I have 10 manually scored recordings which i am using to train the model. When runing the '02_test_state_annotation' these recordings are scored with an average accuracy of 70% with a general range between 60-85%. For one of these recordings however the accuracy is only 35%. I have had a look at the recording to try and figure out why the accuracy is so much lower and it seems some of the channels used are of poor signal quality. Therefore I selected neighbouring electrodes for this recording. However, the accuracy I get out is still exactly the same - why has changing which channels are used, made no difference to performance?

In your paper you were getting accuracies above 90%, do you think it will be a problem if this is lower? Is there any way to try and improve this?

For context, I am using 30 channel EEG electrode arrays which cover the whole skull. I do not have EMG but instead use a composite signal based on a 3-axis accelerometer which I find just as useful/accurate for detecting REM. I guess the preprocessing step would need to be altered for this signal?

paulbrodersen commented 1 week ago

Therefore I selected neighbouring electrodes for this recording. However, the accuracy I get out is still exactly the same - why has changing which channels are used, made no difference to performance?

Did you re-run the pre-processing before testing?

In your paper you were getting accuracies above 90%, do you think it will be a problem if this is lower?

Accuracies below 90% are a clear indicator that something in the pipeline is broken.

For context, I am using 30 channel EEG electrode arrays which cover the whole skull.

Are these surface electrodes, or do they penetrate the skull?

I do not have EMG but instead use a composite signal based on a 3-axis accelerometer which I find just as useful/accurate for detecting REM.

Is the composite signal 1-dimensional?

rjp109 commented 1 week ago

Hello,

You're right, I forgot to re-run the pre-processing. However, unfortunately (and surprisingly) it has made little difference to the accuracy.

These are surface electrodes, placed directly onto the skull.

Yes the composite signal is 1D (basically just using whichever axis has the max value at each time point).