Reset GPU after each epoch

[fshamsafar]

Hi,

I am fine-tuning an “imagenet-resnet-152-dag.mat”. I have compiled MatConvNet-24 using 2×1060 gpus, 64g ram in windows 10. However, after any epoch is finished, I get this error:

Error using Composite/subsref (line 12) Dot invocation is not allowed for Composites, use the functional form instead.

_Error in my_cnn_traindag (line 106) params.getBatch = getBatch(opts, net.meta) ;

_Error in P20_MPII_ResNet (line 99) [net,stats] = my_cnn_traindag(net, imdb, getBatch);

I could not find the source of the error and to bypass the error, I have to reset the gpus after each epoch and continue training the network from the previous epoch where it had left. Is there any setting in MatConvNet to solve the issue?

Thanks.

[fshamsafar]

I found source of the error and I deem it is a bug in MatConvNet1.0-beta24 (I have not used other versions for multiple GPU scenario and maybe this is the case only in version 24). In the “cnn_train_dag.m” m-file, the command line:

if isa(net, 'Composite'), net = net{1} ; end

must be in the for-loop of epoch and not in the outside of the loop. By locating it in the last line of the for-loop, it changes from composite object to a meaningful class that can be processed into the processEpoch for the next epoch. This is the case only when multiple GPUs are used.