search

juglab / cryoCARE_pip

PIP package of cryoCARE
BSD 3-Clause "New" or "Revised" License
25 stars 14 forks source link

F tensorflow/stream_executor/cuda/cuda_dnn.cc:88] Check failed: narrow == wide (-2128347136 vs. 2166620160)checked narrowing failed; values not equal post-conversion #53

Closed rnwokonko closed 9 months ago

rnwokonko commented 10 months ago

For some reason I get this error on the predict step. It doesn't happen for all tomograms, and I can't seem to track down the common issue. Was wondering if anyone has run into this?

My workflow is -> motioncor2 splitsum -> warp to make tilt stacks -> aretomo align/reconstruction (bin4). This current issue is coming from tomograms reconstructed from this data set: https://www.ebi.ac.uk/empiar/EMPIAR-10499/. I am currently using it to test that my new workstation was running smoothly, but have run into this error.

Any insight would be very appreciated.

tibuch commented 9 months ago

Hi @rnwokonko,

I have encounter this issue in the past. Unfortunately I don't have a explanation to what is the root cause for this issue. But it seems to be connected to the tomogram size. I usually managed to get the network to work by increasing n_tiles slightly which reduced the size of the patches that were denoised.

rdrighetto commented 9 months ago

I just encountered the same issue and increasing the number of tiles to [4, 4, 4] made it work.

tibuch commented 9 months ago

Most likely this is clear, but I just want to mention it here again. Any number of tiles works, not only powers of two. So increasing from (4 ,4, 4) to (5, 4, 4) would also be fine.

rnwokonko commented 9 months ago

Just wanted to circle back - it looks like increasing ntiles from [1,1,1] to [4,4,4] worked for my data set. I haven't tried other numbers.

Thanks for the input.