I wanted to change the Implementation of MobilebetV3Small for my use case. so I cloned the keras-applications and tried to run the mobileNetV3 directly from the source. but it had raised errors when I was using GPU
tensorflow Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try to see if a warning log message was printed above.
I tried TPU and CPU and everything was right. so I tried to limit the GPU memory usage manually to solve it as GPU memory was almost full when I got the error. as a result, I had to pick a smaller batch and it didn't work well so I gave up and trained my network with TPU but it took so long.
Summary
there is a bug when you are training your network with GPU
I wanted to change the Implementation of MobilebetV3Small for my use case. so I cloned the keras-applications and tried to run the mobileNetV3 directly from the source. but it had raised errors when I was using GPU
tensorflow Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try to see if a warning log message was printed above.I tried TPU and CPU and everything was right. so I tried to limit the GPU memory usage manually to solve it as GPU memory was almost full when I got the error. as a result, I had to pick a smaller batch and it didn't work well so I gave up and trained my network with TPU but it took so long.Summary
there is a bug when you are training your network with GPU
Environment
Logs or source codes for reproduction
model5=models.Sequential()model5.add(MobileNetV3(stack_fn,1000,input_shape=(480,640,3),model_type='small',alpha=0.75,minimalistic=False,include_top=False,backend=keras.backend))model5.add(layers.GlobalMaxPool2D())model5.add(layers.Flatten())model5.add(layers.Dense(9, activation='tanh'))model5.add(layers.Dense(3))model5.build(input_shape=(1,480, 640, 3))