search

tbepler / topaz

Pipeline for particle picking in cryo-electron microscopy images using convolutional neural networks trained from positive and unlabeled examples. Also featuring micrograph and tomogram denoising with DNNs.
GNU General Public License v3.0
172 stars 64 forks source link

error when reading topaz denoise3d output #67

Closed ssscj closed 3 years ago

ssscj commented 4 years ago

Hi authors,

I use topaz denoise3d to denoise tomogram, and error occurs when I use imod 3dmod to see the output.

  1. It says " ERROR: mrcReadSectionAny -reading data from file. 3dmod :Fatal Error -- while reading image data. System error: Resource temporarily unavaliable " . And when I try to use mrcfile package in python to open the data, it says " ValueError: Expected 2229534720 bytes in data block but could only read 2229534453
  2. the result is wrong, raw tomo(left) and denoised result(right) is here image

Thank you !

tbepler commented 4 years ago

Can you share your tomogram and the exact command you ran so that we can reproduce this?

ssscj commented 4 years ago

Thank you. tomogram can be download directly by this link: https://cloud.tsinghua.edu.cn/f/d66306bf61f44c7aac10/?dl=1 and the command is topaz denoise3d ../tomograms/ts_11_bin4.mrc -o ./ -m unet-3d-20a

ssscj commented 3 years ago

Hi, is there any problem when downloading the tomogram? Maybe I could use another download link.

alexjnoble commented 3 years ago

Hi ssscj,

I just clicked the link and it says 'Sorry, but the requested page could not be found.'

Best, -Alex

ssscj commented 3 years ago

Hi Alex,

Sorry maybe the link was out of date. There are two new links https://drive.google.com/file/d/14EK1hVFJFjY_hHwZBFCEmIc-_q4NF1oT/view?usp=sharing https://cloud.tsinghua.edu.cn/f/608d7c22c21348f7ba3d/?dl=1 Thank you.

Best, chujie

alexjnoble commented 3 years ago

Hi chujie,

I just tried denoising on GPU and I reproduced your error: System error: Resource temporarily unavaliable " when trying to open it with 3dmod. Both EMAN and IMOD report bad statistics, and IMOD reports a bad data block.

I tried normalizing the input fire with EMAN and then denoising creates an mrc that is valid, but when I look at it, it is mostly black with a few cubes.

@tbepler Can you look into the issue with this tomogram filetype/datatype? It looks like it was made with BSoft, which is rarely-used software in my experience.

Best, -Alex

tbepler commented 3 years ago

@alexjnoble Please send me the denoised tomogram.

tbepler commented 3 years ago

Ok, I figured out the problem. @ssscj your tomogram has an extended header. This is pretty unusual and is actually the first tomogram/micrograph I've ever seen that uses the extended header section. Topaz doesn't write the extended header when denoising which causes a parsing error, because programs read part of the tomogram as the extended header, and then the number of bytes in the remaining section is incorrect. You can see this if you look at the file sizes of the raw and denoised tomograms. The denoised one is slightly smaller because the extended header is missing.

tbepler commented 3 years ago

This is now fixed in commit https://github.com/tbepler/topaz/commit/23123cdaf784400f268aba55394512077a9a260b

@ssscj if you pull the newest version of topaz from the master branch, you should be able to denoise your tomograms and have the output be correctly parsed.

ssscj commented 3 years ago

Thanks, it works correctly. I will replace Bsoft with other software in my workflow. Thank you.