search

microsoft / CNTK

Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit
https://docs.microsoft.com/cognitive-toolkit/
Other
17.49k stars 4.3k forks source link

Sequence classification LSTM c# (question) #3796

Closed MihaiHoriaPopescu closed 4 years ago

MihaiHoriaPopescu commented 4 years ago

Hello, I have some questions regard sequence classification using RNN or LSTM. I have used the example for the https://github.com/microsoft/CNTK/blob/release/latest/Examples/TrainingCSharp/Common/LSTMSequenceClassifier.cs to create a classifier for some motion gesture recognition, and create an evaluator as shown in https://github.com/microsoft/CNTK/blob/release/latest/Examples/TrainingCSharp/Common/TestHelper.cs to evaluate my model. I have tuning the parameters to maximise the value of the right label prediction in the result array var actualLabels = outputData.Select(l => l.IndexOf(l.Max())).ToList(); and minimize the value of the other labels. (the value for a good gesture is somewhere at (9/10), where the other values are maximum at (2/3)). The results are great if the gesture represent a good gesture, but the problems comes with all the other sequences that should not be recognized as a trained gesture. So evaluating all the other gesture I would imagine to get all the values in the evaluation array very low, instead I get a prediction as it should be part of the trained gesture with a value of (9/10) as for one of the good gestures. Is it good to use that array as a confidence for the evaluation? I whould expect for all the other gestures that were not trained to get a low confidence for all the trained gesture evaluation since we cannot train all the other infinite gesture as a default gesture. Am I wrong?Is LSTM classification a good approach for my needs? Thanks!

delzac commented 4 years ago

It may not have anything to do with the architecture. Is your dataset balanced? Meaning, are there roughly equal numbers of positive and negative examples?

MihaiHoriaPopescu commented 4 years ago

@delzac I do not understand what you mean negative examples. In the training set I can have only positive examples. All the gestures that were not trained are negative. Example: Features of the gesture Label

I have a balanced training set with same number of session for each label.

delzac commented 4 years ago

I think you might need to study more on machine learning in general first to help yourself out. Its perfectly fine to use LSTM to do what you are doing. The issue here is the setup of your training.

delzac commented 4 years ago

Presumably you have a few classes (gesture). In the event where the gesture is no part of any of the trained gesture, what do you expect the network to do?

Simple example, i have an image classifier trained on dogs and cats. I give an image of a turtle, what output should the network give?

MihaiHoriaPopescu commented 4 years ago

Your example is actualy very good for what I'm trying to archive. I would like to archive an classifier where you can get a classification of cats and dogs, but how can I train the system to not predict all the other images as not dogs and cats? I immagine that you can improve the accuracy giving all the other animals that are not cats and dogs and train with "other animal" label. But if you in the evaluation don't get an animal, what you expect? So what I am trying to archive is a system that classify only the labels that I need with an overfitting, so when I don't get cats or dogs image I get a prediction of cat and dogs, but with low confidence, so you're final result will be cats or dogs with some confidence, and other with lower confidence. This because is impossible to train all the images to archive a good result of "no cat or dog".

delzac commented 4 years ago

In the case of dogs and cats, how it can be handled is that you have 3 class Dog, Cat, Others. The dataset for Others will be random images (animals, buildings, aeroplanes, whatever).

If you only have Dog and Cat, you cannot give low scores to both of them at the same time. All the probabilities must sum to 1.

An alternative method is to build an anomaly detector and filter off out-of-sample data i.e. non dogs and cats before send those images to the dog-cat classifier.

MihaiHoriaPopescu commented 4 years ago

@delzac Ok, thatnk you for the replies. I will try to investigate more for the argument.