search

juglab / n2v

This is the implementation of Noise2Void training.
Other
385 stars 107 forks source link

tensor dimension input error when running 3D N2V #143

Open ywu130 opened 11 months ago

ywu130 commented 11 months ago

Hello, I encountered this error when running the example 3D training for N2V. The error says: Node: 'model/batch_normalization/FusedBatchNormV3' 2 root error(s) found. (0) INVALID_ARGUMENT: input must be 4-dimensional[4,32,64,64,32] [[{{node model/batch_normalization/FusedBatchNormV3}}]] [[gradient_tape/model/batch_normalization_1/FusedBatchNormGradV3/_184]] (1) INVALID_ARGUMENT: input must be 4-dimensional[4,32,64,64,32] [[{{node model/batch_normalization/FusedBatchNormV3}}]] 0 successful operations. 0 derived errors ignored. [Op:__inference_train_function_3323]; It seems it is expecting 2D input, not 3D (5D tensor). And indeed the 2D training example works for me. I am running the example notebook locally. My N2V version: 0.3.2; my tensorflow version: 2.13.0; I am using a M1 macbook. was this error related to version issue? What would you suggest me to do? Thank you!

jdeschamps commented 10 months ago

Hi!

We will try to have a look in the coming days! In the mean time, you could try with older versions of tensorflow. I am pretty sure the notebook was tested with TF2.10 but not with TF2.13.

ywu130 commented 10 months ago

Thank you! I am using a M1 macbook and it only supports TF2.13 to my knowledge. Also I tested the napari-n2v plugins. II used napari 0.4.18 and TF2.13, and napari-n2v 0.1.0. Same issue (tensor dimension error) appeared.

jdeschamps commented 10 months ago

It seems to be a known M1 problem, but there also seem to be solutions. Could you try the suggestion here: https://github.com/juglab/n2v/issues/133?

jdeschamps commented 10 months ago

Any luck?

ywu130 commented 10 months ago

still no luck. I created a new conda environment with tensorflow 2.9.0 and tensorflow-metal 0.5.0. And n2v 0.3.2. I reproduced the error I got earlier (the same as #133 ). 2D works fine still.

veegalinova commented 10 months ago

Seems like a tensorflow-metal specific issue, and also reproduces on M2. Unfortunately, I wasn't able to find a solution yet.

For now, training of 3D model on M1/M2 seems possible only on CPU:

with tf.device("/cpu:0"):
    history = model.train(X, X_val)
Emmakikix commented 3 months ago

I had a similar issue when I ran 2D-RGB and it prompted: "InvalidArgumentError: image must be 3-dimensional[3,64,64,3] [Op:EncodePng]" N2V version: 0.3.3; tensorflow version: 2.7.0; I am using windows 10; This happedened when the training finished the first epoch. I tried running with cpu but it didn't work.