Training failed after data augmentation

[Lawliet121]

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context Add any other context about the problem here.

[Lawliet121]

The data can be trained normally without augmentation. However, after data augmentation with default parameters, loss value will fluctuate greatly and disappear after training for a while.

![Uploading loss=nan.png…]()

[esgomezm]

Hey @Lawliet121 Thank you for your message, could you please specify the notebook and the type of data you are using? Do all the images have the same shape for example, or are the names for the target and source images the same? This will help us solving the issue.

[Lawliet121]

Hey @esgomezm Thank you for you reply! I am using U-net2D to train some single cell images. All the images are single cells with similar shape and the names for the target and source images are the same. All the masks are made of black background with value equal to 0 and white part around the cell with value equal to 1.

[esgomezm]

Hi @Lawliet121

Not sure if this is the problem but I can see that the two images you show have different shape and also that the ratio background / foreground is super low. This should not be a problem but still, it might be that when the notebook crops patches, it discards most of them because of the low ratio. Could you check whether the patches in "/content/img_patches", /content/mask_patches" and "/content/rejected" make sense? Also, after the augmentation, I check it and the differences in the size of the images should not be a problem. Still, could you check that what is being generated in "/content/augment_img" and "/content/augment_mask" makes sense?

Thank you and sorry for the errors,

[ivitoriaUC3M]

Hello! I am having the same issue in notebook _U-Net_2DZeroCostDL4Mic.ipynb . I am using non-round segmented images, even if they kind of look similar, non of the masks are the same. Both image and mask pairs sizes are the same size (2018 x 3825, for instance). However, not all the images (nor masks) are the same size (1549 x 3431, for example). Images and masks have the same name. All the masks are made of black background with value equal to 0 and white part around the cell with value equal to 1. How can I fix this?