Closed dcupolillo closed 5 months ago
Question: Is your code working the "our" d3set files that we provide? For reproducibility reasons: Can you provide me some example d3set files?
Yes, the code works with DeepD3_Training.d3set and DeepD3_Validation.d3set. In few seconds I get the epochs running.
Here you can find a training.d3set
and a validation.d3set
.
Thanks for the help!
Dear Dario,
these were the issues:
Your data has shape (1, 125, 65)
, which is very very small. Therefore, you get an infinite loop, when the stream
would like to generate (1,128,128)
images.
I reduced the input to the training to (1, 32, 32)
. That means, you need to adjust the U-Net as well as the DataGenerators. It is now very small in the bottleneck (1, 2, 2)
which may cause problems. You may need to adjust this.
int16
as dtype is also problematic, please use uint16
. I fixed this by just shifting the data by the minimum.
With these changes, I could make the DeepD3 neural net run and it seems to converge. Results are not tested. If you use a fixed size for training, please remember that inference needs a flexible size (see hints on the DeepD3 website). Training_DeepD3_model.zip
Description
I am attempting to train a model using my own 2D dataset with DeepD3, but the training process hangs when following instructions provided in
Training DeepD3 model.ipynb
, and it seems like it is not engaging the GPU as expected.Images are signed int16 grayscale tiff images, dendritic and spines maks are binary tiff images, as shown in the example images below:
The creation of training data using the
deepd3-training
GUI and the generation of.d3data
files proceed without issues. The generated .d3data is correctly displayed as shown in the screenshot below:Steps to Reproduce issue
deepd3-training
GUI. Place bounding box, enter pixel size, resolution in microns and z step = 0.training.d3set
andvalidation.d3set
filesTraining DeepD3 model.ipynb
Jupyter notebook (running in Anaconda Spyder) up to them.fit
function call.Expected Behavior
Actual Behavior
m.fit
function, with no error messages but also no Epochs progress and inadequate GPU engagement.Environment
Additional Context
Tasks
m.fit
function on the generated d3sets worked as expected in my setup.m.fit
function worked as expected in my setup.