How to use the backward and save function on source code in window?

[ghost]

My plan is making an online(real-time) (deep) learning machine. I think I need the function:backward and save on source code level. although I tried searching about backward, I cant find it.

C++ Window Source code NuGet Package CNTK.GPU 2.1.0 Visual Studio 2015 with Update 3

Help me please. Thank you.

[ebarsoumMS]

What you mean save on the source code level? Each Node have a forward and backward function. Can you explain a little about your planned workflow?

[ghost]

I see. Each node have the function(forward and backward). Yes, I agree. My question is easy.

(https://docs.microsoft.com/en-us/cognitive-toolkit/cntk-library-evaluation-on-windows) (https://github.com/Microsoft/CNTK/blob/release/2.1/Examples/Evaluation/CNTKLibraryCPPEvalCPUOnlyExamples/EvalMultithreads.cpp)

These two link provide evaluation example. There are many types of forward method, but I can't find backward example.

Just want to know how to use the function(backward and save)

[n17s]

Backward is invoked when you call TrainMinibatch. You typically don't need to call backward by yourself. You can look at the documentation of backward for Python. The C++ version is quite similar.

[ghost]

---

edit ( please read next comment)

I did not use the train function(TrainMiniBatch). Only use these functions(createBatch, forward). If the two functions perform similar function(TrainMiniBatch), then It is okay.

I wonder the two functions call the backward function automatically or not.

If yes, how to save the trained network? Also same question(How to save the network? I can not find example.)

If no, how to use the backward and save, Need to example.

Thank you for your answer.

---

edit ( please read next comment)

[ghost]

I find the save function. now, the problem is only one about backward.

@n17s @ebarsoumMS

I think, the function, forward does not perform automatically backward function. So I want to use/call the function, backward. Backward function's inputs are three (BackPropStatePtr, rootGradientValues, backPropagatedGradientValuesForInputs)

Forward function return BackPropStatePtr. But still I can not input 2 parameter rootGradientValues, backPropagatedGradientValuesForInputs. How can I get/set the input?

Is there any simple example to use backward?

---

edit

I find the example file (https://github.com/Microsoft/CNTK/blob/master/Tests/UnitTests/V2LibraryTests/FeedForwardTests.cpp) The example (TestFeedForwardNetworkCreation) gives me some hints. My method is by using load(model file), but the example shows backward by creating. So, I tried modification.

---

code start

GetVariableByName ( evalFunc->Arguments ( ) , L"features" , inputVar ); GetVariableByName ( evalFunc->Outputs ( ) , L"z" , outputVar ); GetVariableByName ( evalFunc->Outputs ( ) , L"ce" , trainingLoss ); inputShape = inputVar.Shape ( ).AppendShape ( { 1, 1 } ); outputShape = outputVar.Shape ( ).AppendShape ( { 1, 1 } ); ValuePtr inputValue = MakeSharedObject ( MakeSharedObject ( inputShape , inputData , true ) ); ValuePtr outputValue; unordered_map<Variable , ValuePtr> outputs = { { outputVar, outputValue } }; auto backpropValue = evalFunc->Forward ( { { inputVar, inputValue } } , outputs , device ); outputValue = outputs [ outputVar ]; NDArrayViewPtr cpuArrayOutput = MakeSharedObject ( outputShape , outputData , false ); cpuArrayOutput->CopyFrom ( *outputValue->Data ( ) );

NDShape outputShape = trainingLoss.Shape ( ); std::vector rootGradientsData ( outputShape.TotalSize ( ) , 1 ); ValuePtr rootGradientValue = MakeSharedObject ( MakeSharedObject ( outputShape , rootGradientsData.data ( ) , rootGradientsData.size ( ) , DeviceDescriptor::CPUDevice ( ) , true ) ); std::unordered_map<Variable , ValuePtr> paramGradients; auto allParams = evalFunc->Parameters ( ); for ( auto iter = allParams.begin ( ); iter != allParams.end ( ); ++iter ) paramGradients [ *iter ] = nullptr; evalFunc->Backward ( backpropValue , { { trainingLoss, rootGradientValue } } , paramGradients );

---

code end

There is maybe one issue about trainingLoss. In the example, trainingLoss's data type is FunctionPtr. But my trainingLoss's data type is Variable.

I think this difference occur error.

---

cntk config file start

command = Train

stderr = "error.txt"

modelPath = "Models/Train.dnn" deviceId = 0

Train = [ action = "train" BrainScriptNetworkBuilder = [

    inputDim = 15960
    outputDim = 11900

    model = Sequential (
        DenseLayer {4096, activation=ReLU, bias=true, initBias=0.1, initValueScale=0.01} : Dropout :
        DenseLayer {4096, activation=ReLU, bias=true, initBias=0.1, initValueScale=0.01} : Dropout :
        LinearLayer {outputDim, bias=true, initBias=0.1, initValueScale=0.01}
    )

    features = Input (inputDim)
    labels = Input (outputDim)
    z = model (features)

    ce = CrossEntropyWithSoftmax (labels, z)
    errs = ClassificationError (labels, z)

    featureNodes = (features)
    labelNodes = (labels)
    criterionNodes = (ce)
    evaluationNodes = (errs)
    outputNodes = (z)
]
SGD = [
    epochSize = 0
    minibatchSize = 1
    learningRatesPerSample = 0.0000001
    maxEpochs = 1
    DropoutRate = 0.5
    numMBsToShowResult = 1
]
reader = [
    readerType = "CNTKTextFormatReader"
    file = "Train.txt"
    input = [
        features = [
            dim = 15960
            format = "dense"
        ]
        labels = [
            dim = 11900
            format = "dense"
        ]
    ]
]

]

---

cntk config file end

Just simple, but I can not make a training code. Sorry, I do not have an understanding of cntk well.

How to fix the code to train well?