search

HasnainRaz / FC-DenseNet-TensorFlow

Fully Convolutional DenseNet (A.K.A 100 layer tiramisu) for semantic segmentation of images implemented in TensorFlow.
MIT License
123 stars 41 forks source link

why it stop in first trian step when i train it in my dataset #7

Closed FENGShuanglang closed 6 years ago

FENGShuanglang commented 6 years ago

image

HasnainRaz commented 6 years ago

The training is happening, the output is printed every 50 steps, so it might take some time until you see the next step, you can modify the train function to remove this 50 step output condition as needed.

FENGShuanglang commented 6 years ago

yes,you are right!,but i think it is so slow for me ,It takes about 20 seconds to process a picture，I wonder if it is normal？I have used GPU

HasnainRaz commented 6 years ago

The tiramisu is a deep network, and it's memory consumption is quadratic because of how tensorflow computes gradients. Also, your images are high resolution. Combined all these facts, it will be slow. You can either try reducing the dense blocks, number of layers or your image size to make it faster.

If you have access to a better GPU (V100s), that should also help, but I realize not everyone does.

FENGShuanglang commented 6 years ago

OK,thank you very much ,but I wonder Why does the loss not converge? I am doing multi-class segmentation. image

HasnainRaz commented 6 years ago

60 iterations is too low, the Tiramisu is known to be difficult to optimize, see, try training it longer.