When running FCN-16 or FCN-32 with num_classes > ~700 or > ~350 respectively, Tensorflow complains the following and crashes:
ValueError: Cannot create a tensor proto whose content is larger than 2GB.
It fails because the deconv filter is > 2gb with this number of classes (64 x 64 x 350 x 350 in the case of FCN-32) and TF rejects tensors that big. Is there a way to reconfigure the deconv filter/layer so that it doesn't have to allocate all of this at once? My first thought would be to use a generator but tensorflow.py_func() doesn't support generator functions and I haven't found any documentation hinting about the possibility of this.
Note, I tried splitting up the deconv up into several smaller operations, but it turns out TF Graph has a limit of 2gb as well.
Edit: I was able to fix this by splitting the deconv operating into 20 smaller deconv operations, then concating results and running argmax. This produced significantly worse results than just reducing # output classes to 20 and is significantly more computationally expensive.
When running FCN-16 or FCN-32 with num_classes > ~700 or > ~350 respectively, Tensorflow complains the following and crashes:
It fails because the deconv filter is > 2gb with this number of classes (64 x 64 x 350 x 350 in the case of FCN-32) and TF rejects tensors that big. Is there a way to reconfigure the deconv filter/layer so that it doesn't have to allocate all of this at once? My first thought would be to use a generator but tensorflow.py_func() doesn't support generator functions and I haven't found any documentation hinting about the possibility of this.
Note, I tried splitting up the deconv up into several smaller operations, but it turns out TF
Graphhas a limit of 2gb as well.Edit: I was able to fix this by splitting the deconv operating into 20 smaller deconv operations, then
concating results and runningargmax. This produced significantly worse results than just reducing # output classes to 20 and is significantly more computationally expensive.