Open alienatorZ opened 5 years ago
I'm having the same issue.
/cc @klevis
Hi,
May this could help ? https://github.com/eclipse/deeplearning4j/issues/7841
Transform both operands must have same data type; Expected: [DOUBLE]; Actual: [FLOAT]
can be resolved with combination.muli(NOISE_RATION).addi(content.muli(1 - NOISE_RATION).castTo(DataType.DOUBLE));
at line 194.
However, after the change is made, I see a different exception which seems more serious:
Exception in thread "main" java.lang.IllegalStateException: Cannot perform backprop: Dropout mask array is absent (already cleared?)
at org.nd4j.base.Preconditions.throwStateEx(Preconditions.java:641)
at org.nd4j.base.Preconditions.checkState(Preconditions.java:268)
at org.deeplearning4j.nn.conf.dropout.Dropout.backprop(Dropout.java:167)
at org.deeplearning4j.nn.layers.AbstractLayer.backpropDropOutIfPresent(AbstractLayer.java:302)
at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.backpropGradient(ConvolutionLayer.java:192)
at org.deeplearning4j.examples.styletransfer.NeuralStyleTransfer.backPropagate(NeuralStyleTransfer.java:292)
at org.deeplearning4j.examples.styletransfer.NeuralStyleTransfer.backPropagateStyles(NeuralStyleTransfer.java:170)
at org.deeplearning4j.examples.styletransfer.NeuralStyleTransfer.transferStyle(NeuralStyleTransfer.java:145)
at org.deeplearning4j.examples.styletransfer.NeuralStyleTransfer.main(NeuralStyleTransfer.java:126)
Any thoughts on how to fix that? Does this mean the layers list/order is incorrect?
Ok, I kind of was able to run the style transfer example, but at this point I'm not sure if it is mathematically correct/efficient. Here is the patch content / intellij 2019.2:
Index: dl4j-examples/src/main/java/org/deeplearning4j/examples/styletransfer/NeuralStyleTransfer.java
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- dl4j-examples/src/main/java/org/deeplearning4j/examples/styletransfer/NeuralStyleTransfer.java (revision 20225a8a70b8932677538d55be55ccacafa01333)
+++ dl4j-examples/src/main/java/org/deeplearning4j/examples/styletransfer/NeuralStyleTransfer.java (date 1567049714186)
@@ -19,11 +19,13 @@
import org.datavec.image.loader.NativeImageLoader;
import org.deeplearning4j.examples.download.DownloaderUtility;
import org.deeplearning4j.nn.api.Layer;
+import org.deeplearning4j.nn.conf.dropout.IDropout;
import org.deeplearning4j.nn.graph.ComputationGraph;
import org.deeplearning4j.nn.workspace.LayerWorkspaceMgr;
import org.deeplearning4j.zoo.PretrainedType;
import org.deeplearning4j.zoo.ZooModel;
import org.deeplearning4j.zoo.model.VGG16;
+import org.nd4j.linalg.api.buffer.DataType;
import org.nd4j.linalg.api.ndarray.INDArray;
import org.nd4j.linalg.dataset.api.preprocessor.DataNormalization;
import org.nd4j.linalg.dataset.api.preprocessor.VGG16ImagePreProcessor;
@@ -145,7 +147,7 @@
INDArray backPropContent = backPropagateContent(vgg16FineTune, activationsContentMap, activationsCombMap);
INDArray backPropAllValues = backPropContent.muli(ALPHA).addi(styleBackProb.muli(BETA));
adamUpdater.applyUpdater(backPropAllValues, iteration, 0);
- combination.subi(backPropAllValues);
+ combination.subi(backPropAllValues.castTo(DataType.DOUBLE));
log.info("Total Loss: " + totalLoss(activationsStyleMap, activationsCombMap, activationsContentMap));
if (iteration % SAVE_IMAGE_CHECKPOINT == 0) {
@@ -190,7 +192,7 @@
INDArray content = LOADER.asMatrix(new File(dataLocalPath,CONTENT_FILE));
IMAGE_PRE_PROCESSOR.transform(content);
INDArray combination = createCombineImageWithRandomPixels();
- combination.muli(NOISE_RATION).addi(content.muli(1 - NOISE_RATION));
+ combination.muli(NOISE_RATION).addi(content.muli(1 - NOISE_RATION).castTo(DataType.DOUBLE));
return combination;
}
@@ -288,7 +290,17 @@
for (int i = startFrom; i > 0; i--) {
Layer layer = vgg16FineTune.getLayer(ALL_LAYERS[i]);
- dLdANext = layer.backpropGradient(dLdANext, LayerWorkspaceMgr.noWorkspaces()).getSecond();
+ org.deeplearning4j.nn.conf.layers.Layer conf = (org.deeplearning4j.nn.conf.layers.Layer)layer.getConfig();
+
+ try {
+
+ ((org.deeplearning4j.nn.conf.layers.Layer) layer.getConfig()).setIDropout(null);
+
+ dLdANext = layer.backpropGradient(dLdANext, LayerWorkspaceMgr.noWorkspaces())
+ .getSecond();
+ }catch (Exception e){
+ log.error("Error {} => {}", layer.getConfig().getClass().getSimpleName(), layer.getConfig().getLayerName(), e);
+ }
}
return dLdANext;
}
@AlexDBlack What do you think?
This issue still open. I was able to do style transfer just fine with the diff fix, even on other works(custom drawings). So, should I create an MR for my fix from above?
Thank you vera-dobryanskaya
Issue Description
Please describe your issue, along with:
expected behavior Run until execution finished.
encountered behavior Error stops execution.
Stack Trace: Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.intellij.rt.execution.CommandLineWrapper.main(CommandLineWrapper.java:66) Caused by: java.lang.RuntimeException: NativeOps::execPairwiseTransform both operands must have same data type; Expected: [DOUBLE]; Actual: [FLOAT] at org.nd4j.nativeblas.Nd4jCuda$NativeOps.execPairwiseTransform(Native Method) at org.nd4j.linalg.jcublas.ops.executioner.CudaExecutioner.invoke(CudaExecutioner.java:1420) at org.nd4j.linalg.jcublas.ops.executioner.CudaExecutioner.exec(CudaExecutioner.java:635) at org.nd4j.linalg.api.ndarray.BaseNDArray.addi(BaseNDArray.java:3818) at org.nd4j.linalg.api.ndarray.BaseNDArray.addi(BaseNDArray.java:3788) at org.deeplearning4j.examples.styletransfer.NeuralStyleTransfer.createCombinationImage(NeuralStyleTransfer.java:192) at org.deeplearning4j.examples.styletransfer.NeuralStyleTransfer.transferStyle(NeuralStyleTransfer.java:132) at org.deeplearning4j.examples.styletransfer.NeuralStyleTransfer.main(NeuralStyleTransfer.java:124) ... 5 more
Version Information
Please indicate relevant versions, including, if relevant:
Contributing
If you'd like to help us fix the issue by contributing some code, but would like guidance or help in doing so, please mention it!