Errors when trying to predict after training

[SirishNutakki]

We successfully trained a model on an 11gb graphics card, but when trying to run predictions, we ran into a CUDA out of memory error.

We then tried cropping the images to try to fix this error. We tried predicting with the original model and a new model trained on the cropped images but ran into an error stating that the size of the two tensors in fnet_nn_3d_params.py line 75 don't match.

We printed the sizes of the two tensors and found that they are the same size at the point where the error occurs.

Do you have any advice on how to fix these issues?

[alxndrkalinin]

Check that your image sizes in X and Y are divisible by 16. You can crop or pad if they're not.

Basic solution: add following code before return statement in the item_from_dataset function (assumes X,Y sizes are even)

    # crop XY to the nearest size divisible by 16
    if signal.shape[2] % 16 != 0:
        signal = signal[:,:,signal.shape[2]%16//2:0-signal.shape[2]%16//2,:]
        target = target[:,:,target.shape[2]%16//2:0-target.shape[2]%16//2,:]
    if signal.shape[3] % 16 != 0:
        signal = signal[:,:,:,signal.shape[3]%16//2:0-signal.shape[3]%16//2]
        target = target[:,:,:,target.shape[3]%16//2:0-target.shape[3]%16//2]

[SirishNutakki]

Thanks for the help. The solution you suggested fixed the error, but it leads to a different error with the number of channels.

[alxndrkalinin]

What are the sizes of your images at training and at prediction? Fnet also requires the minimum of 32 slices in Z.

[SirishNutakki]

We're using the images downloaded from download_and_train.py and cropping them in the x and y dimensions, so they do have at least 32 slices in Z. The model was trained on the original uncropped images and the size of the cropped images are 688x464.