microsoft / CNTK

Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit
https://docs.microsoft.com/cognitive-toolkit/
Other
17.49k stars 4.3k forks source link

Value goes invalid when using TestMinibatch #3859

Open viraptor opened 2 years ago

viraptor commented 2 years ago

I'm running an issue when calling trainer.PreviousMinibatchLossAverage() after a TestMinibatch run:

var minibatchDataTest = minibatchTest.GetNextMinibatch(10, device);
var mbData = new UnorderedMapVariableMinibatchData();
mbData.Add(features, minibatchDataTest[featureTestStreamInfo]);
mbData.Add(labels, minibatchDataTest[labelTestStreamInfo]);
var testRes = trainer.TestMinibatch(mbData, device);
Console.WriteLine($"test loss {trainer.PreviousMinibatchLossAverage()}");  // this raises exception

The printed stack is:

[CALL STACK]
    > CNTK::Internal::  UseSparseGradientAggregationInDataParallelSGD
    - CNTK::Value::  Create
    - CNTK::Internal::  UseSparseGradientAggregationInDataParallelSGD
    - CNTK::TrainingParameterSchedule<unsigned __int64>::  Transform
    - CNTK::Trainer::  PreviousMinibatchLossAverage
    - CSharp_CNTK_Trainer_PreviousMinibatchLossAverage
    - 00007FFDE9FB695B (SymFromAddr() error: The specified module could not be found.)

I'm not even sure which value it is, since both minibatchDataTest[featureTestStreamInfo] and minibatchDataTest[labelTestStreamInfo] are valid both before and after the TestMinibatch call. (as reported by data.IsValid)