SVM Training error is: invalidModelType Error

[heuism]

i ran the code which is svm.trainClassifier(...) however they always say that there is an error so i use the catch and print out what type of error is

do {
//            try svm.train(trainData)
    try svm.trainClassifier(trainData)
        }
        catch let error as Error {
            print("SVM Training error is: \(error)")
        }

that and the result is 'invalidModelType' even though i set the type is

let svm = SVMModel(problemType: .c_SVM_Classification, kernelSettings: KernelParameters(type: .linear, degree: 0, gamma: 0.5, coef0: 0.0))

according to the instruction.

Thanks for your help

[KevinCoble]

Classification uses an integer label for the output. When creating a data set for training of a classifier you define a set of integers for each class an input vector can belong to. See the main page of the manual in the section 'Using a Classification Algorithm' for an example. I suspect you are setting the SVMModel to a classification type (c_SVM_Classification), but sending it a regression data set (with doubles for outputs based on your last post to issue #6). This results in the error thrown. Explicit code for using support vectors with classification can be found in the SupportVectorMachine playground in the project as well.

[heuism]

Thanks for your response. But i am sure that i just copy as your manual to test before doing anything

Here is the image that i screenshot the code for you to see

And as i go into your code i can see the part of trainClassifier that:

Which say that if they type is not c or v_SVM_Classification it will throw error as invalidModelType

rather than invalidType of trainingData type.

I hope that i explained my problem clearly :)

Thanks for your help

[KevinCoble]

for now, try just svm.train(trainData) instead of svm.trainClassifier(trainData) - that is what the XCTest example code is using. I will try to investigate what is up with svm.trainClassifier

[heuism]

Hi Kevin,

Thanks for your suggestion, i will try this way!

You did a great job building this lib.

Have a good day, mate

[KevinCoble]

I just updated the SVMExtensions.swift file to address this issue. Get the update and see if it works for you.

P.S. I'll try to look at the 'embedding' issue soon.

[heuism]

@KevinCoble, it passed the phase of training but the phase of classify data is got the same issue Error having SVM calculate results, with Error is: invalidModelType

public func classify(_ testData: MLClassificationDataSet) throws
    {
        //  Verify the SVMModel is the right type
        if type != .c_SVM_Classification || type != .ν_SVM_Classification { throw SVMError.invalidModelType }

        //  Verify the data set is the right type
        if (testData.dataType != .classification) { throw DataTypeError.invalidDataType }
        if (supportVector.count <= 0) { throw MachineLearningError.notTrained }
        if (testData.inputDimension != supportVector[0].count) { throw DataTypeError.wrongDimensionOnInput }

        //  Put the data into a DataSet for SVM (it uses a DataSet so that it can be both regressor and classifier)
        if let data = DataSet(dataType: .classification, withInputsFrom: testData) {

            //  Predict
            predictValues(data)

            //  Transfer the predictions back to the classifier data set
            for index in 0..<testData.size {
                let resultClass = try data.getClass(index)
                try testData.setClass(index, newClass: resultClass)
            }
        }
        else {
            throw MachineLearningError.dataWrongDimension
        }
    }

What should i do to fix the same issue as above? i saw you have

if type != .c_SVM_Classification && type != .ν_SVM_Classification { throw SVMError.invalidModelType } in the trainClassifier. So i tried the same thing changing "||" to "&&" in the classify method but it doesn't work.

Thanks @KevinCoble

[heuism]

Dear @KevinCoble, i fixed it by going into your real code and change it so the issue about "invalidModelType is disappeared". Another issue appears because of that. When i try to data.getClass(index), the code say index out of range so i try to print out the class of the first one

//  Put the data into a DataSet for SVM (it uses a DataSet so that it can be both regressor and classifier)
        if let data = DataSet(dataType: .classification, withInputsFrom: testData) {

            //  Predict
            predictValues(data)

            try print("The class is: \(data.getClass(0))")

            //  Transfer the predictions back to the classifier data set
            for index in 0..<testData.size {
                let resultClass = try data.getClass(index)
                try testData.setClass(index, newClass: resultClass)
            }
        }
        else {
            throw MachineLearningError.dataWrongDimension
        }

But still it shows that

the class is 0 values.

Can you help me with this?

Thanks a lot @KevinCoble

[KevinCoble]

When there are more than two class labels in an SVM model there is a requirement to have storage of a 'fit' value for each discrimination between pairs of class labels. To achieve this, make the prediction data type '.realAndClass' rather than just 'classification'. The model will allocate a real array for storage of the discrimination values and attach to each point. They are then used to determine which class matches best, and then the class label is assigned in the data set.

Look in the SVMTests.swift file for an example if needed. I may need to add something to the generic 'classification' examples in the manual to make this more clear. Other classification models do not need this (I had to to it in Support-Vector-Machines as the code was a port of another C++ library, and it was required to fit it into swift protocols I am using)

[heuism]

@KevinCoble, thanks for your instruction, i tried the same thing you said but the error is still there, but when i copy the code from the test as example and alternate a bit, the classification works.

Thanks for your hard work, i think i just got one more question then i can close this issue that why you put the ".classification" there @ first place? would it be better to use ".realAndClass" for 2 and above? instead of dividing? And as you state there ARE another classifiers as well? Are they listed in your manual as well?

Thanks a log @KevinCoble

[KevinCoble]

Other classifiers will only need the .classification type - it was just the port of the C++ library that required me to make the .realAndClass type for storage of the discrimination values (SVM's can only discriminate between two classes, so if there are more than two, you discriminate between each possible pair of two of the available labels, and determine which label wins the most test. Storage is needed for each point to be classified to remember the scores. Other classifiers (like Neural Networks) could also use the .realAndClass type to output confidence levels for each class, so it may be used more later).

Other classifiers in the framework are the NeuralNetwork and LogisticRegression classes

[heuism]

Dear @KevinCoble, i just try to do the classification but it says index out of range, which then made me to have a look into your code. I can see the part of predicting values that:

case .c_SVM_Classification, .ν_SVM_Classification:
                    print("Predict With multiple class")
                    //  Get the kernel value for each support vector at the input value
                    var kernelValue: [Double] = []
                    for sv in 0..<totalSupportVectors {
                        kernelValue.append(Kernel.calcKernelValue(kernelParams, x: inputs, y: supportVector[sv]))
                    }

                    //  Allocate vote space for the classification
                    var vote = [Int](repeating: 0, count: numClasses)

                    //  Initialize the decision value storage in the data set
                    var decisionValues: [Double] = []

                    //  Get the seperation info between each class pair
                    var permutation = 0
                    for i in 0..<numClasses {
                        for j in i+1..<numClasses {
                            var sum = 0.0
                            for k in 0..<supportVectorCount[i] {
                                sum += coefficients[j-1][coeffStart[i]+k] * kernelValue[coeffStart[i]+k]
                            }
                            for k in 0..<supportVectorCount[j] {
                                sum += coefficients[i][coeffStart[j]+k] * kernelValue[coeffStart[j]+k]
                            }
                            sum -= ρ[permutation]
                            decisionValues.append(sum)
                            permutation += 1
                            if (sum > 0) {
                                vote[i] += 1
                            }
                            else {
                                vote[j] += 1
                            }
                        }
                    }
                    print("Before setOutput - with catch")
                    do {
                        try data.setOutput(index, newOutput: decisionValues)
                    }
                    catch {
                        print("Catch and break")
                        break
                    }

                    //  Get the most likely class, and set it
                    var maxIndex = 0
                    for index in 1..<numClasses {
                        if (vote[index] > vote[maxIndex]) { maxIndex = index }
                    }
                    print("Test to here!")
                    try data.setClass(index, newClass: labels[maxIndex])
                    print("Can it make here?")
                    break

The print() code is what i added in to debug it. And then i found out that the code when it

 do {
                        try data.setOutput(index, newOutput: decisionValues)
                    }
                    catch {
                        print("Catch and break")
                        break
                    }

has an error and that is why it break the code there

Catch and break with error : wrongDimensionOnOutput

, which make the latter codes which is:

var maxIndex = 0
                    for index in 1..<numClasses {
                        if (vote[index] > vote[maxIndex]) { maxIndex = index }
                    }
                    print("Test to here!")
                    try data.setClass(index, newClass: labels[maxIndex])
                    print("Can it make here?")

couldn't be executed. I dont know why do we need to setOutput when we just classifying the classes? And i think we didn't set the Output Dimension that is why it throws i guess?

Thanks for your help a lot

[heuism]

i have a look @ the code and it seems like the "decision values" is what put into the setOutput

print("The Decision Value is :\(decisionValues) with the size: \(decisionValues.count)")
                    print("Before setOutput - with catch")
                    do {
                        try data.setOutput(index, newOutput: decisionValues)
                    }

which i am not too sure why the "decision values" affect the classification result.

Hope my work helps your justification for explaining the thing.

Thanks a lot @KevinCoble

[KevinCoble]

If you look at the testThreeStateClassificationWithProtocol() test in the SVMTests.swift file, you will see the test data allocated with the following line.

    let testData = DataSet(dataType: .realAndClass, inputDimension: 2, outputDimension: 3)

Note that the output dimension is set to 3. This must match the number of classification labels in the training set. As I explained earlier, the SVM model needs storage for each class label to store the results of the tests of each pair of classes that are checked (SVM can only discriminate between two classes, so more than two classes are done as permutations - 3 classes require 3 tests for the three pair combinations, 4 classes require 6 tests, 5 classes requires 10, etc.). The results of these pair-tests are accumulated in the 'real' section of the 'realAndClass' data set - requiring one value for each class.

Does your allocation of at the test data set have the right size (match the number of class labels)?

I should add a check/throw for that, so that the error is less ambigiuous.

[heuism]

oh so the outputDimension is equal to the number of total training labels, interesting. I thought i always be 1 cause it will always return 1 value only. Thanks for your response. In that case every thing is done right. However, what is the role of the output array in the case we do the classification? Thanks for your time.

[KevinCoble]

The output array stores the 'scores' for each pair-wise test done to classify an input (please see my previous comment).

As an example, with three class labels the SVM model, for each input vector, compares the input vector against class 1 with class 2 and stores the results. It then compares compares the input against class 2 with class 3 and stores the result. It then uses the input vector to decide the winner between class 1 and class 3. After all these comparisons are done, the class that 'won' the most comparisons is the 'winner', and that input vector is assigned that class label. As stated before, with 4 class labels there will be 6 comparisons, with 5 labels there are 10 comparisons, etc.

The output 'real' vector stores the 'scores' - the number of individual 2-class comparisons each class label 'won'. A temporary vector could be used, but the base library I used thought some people may want to see the individual results, to see how 'strongly' the class label won over the other classes - so the data is attached to the test vectors so they can be examined later if desired.

[heuism]

Oh so that is the number of testing instead of number of labels :) so for example if i got 7 labels it would be 21 :) that is interesting :)

Thanks for your helps. That issue has been cleared 👍

[KevinCoble]

No, it is sized for the number of labels!

Hopefully this explanation helps:

If you have 4 labels in training, for each input vector you test with the following will occur:

The 4 'reals' in the test dataset (for the input vector) will be set to 0. The SVM will compare the input vector with class label 1 and class label 2. Let's assume class label 1 was the winning label in that comparison. Then the 'real' value for the class label 1 is incremented. The SVM will then compare the input vector with class label 1 and class label 3. Let's assume class label 3 was the winning label in that comparison. Then the 'real' value for the class label 3 is incremented. The SVM will then compare the input vector with class label 1 and class label 4. Let's assume class label 1 was the winning label in that comparison. Then the 'real' value for the class label 1 is incremented again (it is now at 2). SVM compares class label 2 with 3 - incrementing the winner's 'real' value SVM compares class label 2 with 4 - incrementing the winners 'real' value SVM compares class label 3 with 4 - incrementing the winners 'real' value

Now that all the comparisons are made, the label with the largest 'real' value is the class label the input vector will be assigned.

[heuism]

really? Cause when i read the code is like this, sorry if my noob bothers you.

for i in 0..<numClasses {
                        for j in i+1..<numClasses {
                            var sum = 0.0
                            for k in 0..<supportVectorCount[i] {
                                sum += coefficients[j-1][coeffStart[i]+k] * kernelValue[coeffStart[i]+k]
                            }
                            for k in 0..<supportVectorCount[j] {
                                sum += coefficients[i][coeffStart[j]+k] * kernelValue[coeffStart[j]+k]
                            }
                            sum -= ρ[permutation]
                            decisionValues.append(sum)
                            permutation += 1
                            if (sum > 0) {
                                vote[i] += 1
                            }
                            else {
                                vote[j] += 1
                            }
                        }
                    }

the decisionValues count at the end of the for loop if the numberofLabels is 4 then the count will be 6.

After that the code will do:

try data.setOutput(index, newOutput: decisionValues)

which is defined as:

 open func setOutput(_ index: Int, newOutput : [Double]) throws
    {
        //  Validate the data
        if (dataType == .classification) { throw DataTypeError.dataWrongForType }
        if (index < 0) { throw  DataIndexError.negative }
        if (index > inputs.count) { throw  DataIndexError.indexAboveDimension }
        if (newOutput.count != outputDimension) { throw DataTypeError.wrongDimensionOnOutput }

        //  Make sure we have outputs up until this index (we have the inputs already)
        if (index >= outputs!.count) {
            while (index > outputs!.count) {    //  Insert any uncreated data between this index and existing values
                outputs!.append([Double](repeating: 0.0, count: outputDimension))
                if (dataType == .realAndClass) { classes!.append(0) }
            }
            //  Append the new data
            outputs!.append(newOutput)
            if (dataType == .realAndClass) { classes!.append(0) }
        }

        else {
            //  Replace the new output item
            outputs![index] = newOutput
        }
    }

so when the newOutput.count (which is the decisionValues.count) is different from the outputDimension it would throw the error there:

if (newOutput.count != outputDimension) { throw DataTypeError.wrongDimensionOnOutput }

with the outputDimension is set as number of test the error wont be thrown when we init the DataSet

Thanks for your help.

I hope i understand it right.

[KevinCoble]

I believe you are correct. I was confusing the 'decisionValue' array with the 'vote' array. As the vote array can be easily recreated if you have the decisionValue array, but not vice versa, that is the one that is returned if you want to look at it. Most people only care about the class label, not the internal values - but they can give insight as to how 'confident' the decision for that label is.

I apologize for the confusion. The SVM model class is the one piece of the framework that I ported from another work, rather than being my own - so I am less knowledgable as to its inner workings.

[heuism]

@KevinCoble, you are doing a good job porting this thing. I can not even imagine how much time you spent on this. Thanks for your time answering my question and clarifying my noob 👍. I will close this thread now then.