Current performances

[dimstudio]

Dataset: CPR_experiment Target: classRate loss: -92.4418 - accuracy: 0.3706 ('Test accuracy:', 0.3705954) ('Test loss:', -92.46426220102414)

Target: classDepth loss: -65.9041 - accuracy: 0.6015 ('Test accuracy:', 0.6014581) ('Test loss:', -50.17492101954371)

Target: classRelease loss: 1.4347 - accuracy: 0.7242 ('Test accuracy:', 0.7241798) ('Test loss:', 1.718166751821015)

Dataset: CPR_test Target: classRate 0s 173us/sample - loss: 0.8667 - accuracy: 0.7978 ('Test accuracy:', 0.7978142) ('Test loss:', 1.2019080369850326)

Target: classDepth loss: 1.1676 - accuracy: 0.7049 ('Test accuracy:', 0.704918) ('Test loss:', 1.364527590939256)

Target: classRelease loss: 1.1262 - accuracy: 0.7049 ('Test accuracy:', 0.704918) ('Test loss:', 1.290114411565124)

Target: armsLocked + bodyWeight loss: 0.1122 - accuracy: 0.9781 ('Test accuracy:', 0.9781421) ('Test loss:', 0.09334265835623924)

Target: armsLocked loss: 0.3258 - accuracy: 0.9617 ('Test accuracy:', 0.96174866) ('Test loss:', 0.15740652211631337)

Target: bodyWeight loss: 0.1594 - accuracy: 0.9781 ('Test accuracy:', 0.9781421) ('Test loss:', 0.1251428161233745)

Dataset: Tennistable_strokes batch size: (240, 50, 6) Target: correct_strokes (with Kinect) loss: 0.7990 - accuracy: 0.7500 ('Test accuracy:', 0.75) ('Test loss:', 0.8324083089828491)

Target: correct_strokes (without Kinect) loss: 0.8000 - accuracy: 0.8000 ('Test accuracy:', 0.8) ('Test loss:', 0.7873132824897766)

[dimstudio]

The first performance (The Dataset: CPR_experiment Target: classRate) with Test accuracy:, 0.3705954 is bad. Why is that?

[HansBambel]

Could be because there is not enough data to properly classify it. In one file for example were only 5 of samples with classRate = 0 and 208 with classRate = 1.

Then he just learns to put always a 1 there, due to the lack of counter samples.

[dimstudio]

The class distribution doesn't look so bad honestly

[HansBambel]

Maybe have a look at the distribution in the test set. If there is a high portion of classRate = 0 this could explain the low accuracy when the model basically neglects this class.

[HansBambel]

I just realized that we are using binary-cross-entropy as loss for a target that has 3 possible values. That's also why the loss is negative which should not be the case and was bugging me the whole time.

[dimstudio]

I just realized that we are using binary-cross-entropy as loss for a target that has 3 possible values. That's also why the loss is negative which should not be the case and was bugging me the whole time.

That makes lot of sense. These two targets have 3 classes indeed.

[dimstudio]

Hi, a new TableTennis dataset is available (check Skype)

[dimstudio]

New CPR_feedback dataset is ultimated. This is the ultimate dataset we should use for the CPR experiment.

10 participants x 4 sessions = 40 sessions in total

target_classes = ['classRelease' , 'classDepth', 'classRate', 'armsLocked', 'bodyWeight']

Please note: made minor changes to data_helper.py

[dimstudio]

New CPR_feedback dataset is ultimated. This is the ultimate dataset we should use for the CPR experiment.

10 participants x 4 sessions = 40 sessions in total

target_classes = ['classRelease' , 'classDepth', 'classRate', 'armsLocked', 'bodyWeight']

Please note: made minor changes to data_helper.py

The new dataset has unbalanced classes

I had to switch to binary classification for all targets creating CPR_feedback_binarydataset.

classRelease 0 0.39 1 0.61 classDepth 0 0.75 1 0.25 classRate 0 0.4 1 0.6 armsLocked 0 0.28 1 0.72 bodyWeight 0 0.23 1 0.77

Maybe downsampling feature is needed?

[HansBambel]

With this dataset I achieve 93% accuracy (with MSE loss or binary cross entropy), but as you mentioned this could be due to the imbalanced classes.

[dimstudio]

Did you make sure to use the CPR_feedback_binary dataset? The best fix would be to add a function that undersamples the dataset automatically. One quick fix is to delete some sessions (and under-sample the dataset manually). I can send you an updated version of the dataset.

[HansBambel]

Yes, I used that one.

[dimstudio]

Update: I did not downsample, some people say that you create a biased classifier if you do that (source).

I discovered an error in the dataset, which I fixed it, I send you the updated one that scores Accuracy: 0.92399 Precision: 0.93204 Recall: 0.93824

[HansBambel]

Nice! Is the online classification working or do I need to change/add something?

[dimstudio]

Not yet, needs some debugging, it's a quite complicated setup....

[dimstudio]

The system is up and running. However, I notice that the unbalanced classes influence the results, for example, classDepth is mainly predicted as 0 and bodyWeight and armsLocked are always predicted as 1.

I reconsidered downsampling and I thought that since the dataset was already biased since it was created (the participants made on purpose the mistakes) it does not represent a "natural" distribution of the mistakes. So I believe that we could try to downsample the dataset to make the classes even. What do you think? Is there a way to easily do this when splitting the dataset?

[HansBambel]

Yes, I think downsampling would be a good idea. I am not sure about how to do this yet, but I can get to it tomorrow probably. Are you in the office tomorrow?

[dimstudio]

Yes I am at the office, let's discuss this tomorrow.