[BUG] ctfphaseflip - The defocus file tomogram_001.defocus is non-existent or empty

[wlugmayr]

Describe the bug

When running the tutorial ribosome pipeline with TomoBEAR-0.2.0 binary the following error message(s) appears in 8_BatchRunTomo_1/tomogram_001/batchruntomo.log and in 8_BatchRunTomo_1/tomogram_001/BatchRunTomo.log (same name but different size?). Despite the error the job makes a SUCCESS file.

Project JSON file

ribosome_empiar_10064_dynamo.json

Error message

Batchruntomo started on data set tomogram_001, Thu Mar 9 14:43:49 2023 [brt11] Reached step 10

Correcting for CTF with Ctfphaseflip (running ctfcorrection.com) [brt2] processchunks ERROR: ctfcorrection.com has given processing error 1 times - giving up ERROR: ctfphaseflip - The defocus file tomogram_001.defocus is non-existent or empty - did you save in ctfplotter?

ABORT SET: An error occurred running ctfcorrection.com

Additional context

the file exists as link $ ls -l tomogram_001.defocus ... tomogram_001.defocus -> ../../7_GCTFCtfphaseflipCTFCorrection_1/tomogram_001/tomogram_001.defocus $ cat tomogram_001.defocus 1 0 0. 0. 0 3

[wlugmayr]

When restarting the pipeline it gives the following message:

INFO: Skipping processing step 8 (BatchRunTomo_2)! INFO: Skipping pipeline step (BatchRunTomo) due to availability of a SUCCESS file! Error using fprintf Invalid file identifier. Use fopen to generate a valid file identifier.

Error in LocalPipeline/execute (line 448)

Error in runPipeline (line 105)

MATLAB:FileIO:InvalidFid

[wlugmayr]

The error also appears with the latest develop git checkout and running TomoBEAR in interactive MATLAB.

[MishaKudryashev]

Thank you, we will look at it and will try to fix it with the new release!

[wlugmayr]

Since I cannot install imod 4.10, I have tested it with 4.9.12 and 4.11.24 and this error seems to come from imod.

[wlugmayr]

I tried cd output/8_BatchRunTomo_1/tomogram_001 subm ctfplotter.com

$ cat ctfplotter.log

Entries to program ctfplotter InputStack = tomogram_001.st AngleFile = tomogram_001.tlt AxisAngle = -5.0 DefocusFile = tomogram_001.defocus PixelSize = 0.262 Voltage = 300 SphericalAberration = 2.7 AmplitudeContrast = 0.07 BidirectionalNumViews = 31 ExpectedDefocus = 0.0 AngleRange = -20.0 20.0 PSResolution = 101 TileSize = 256 DefocusTol = 200 LeftDefTol = 2000.0 RightDefTol = 2000.0 End of entries

No MaxCacheSize is specified, set to default 192720 Megs ERROR: ctfplotter - Invalid expected defocus, it must be >0 ERROR: ctfplotter -StandardInput: exited with status 1

$ cat tomogram_001.defocus 1 0 0. 0. 0 3

[ArtsemiY]

Dear @wlugmayr

please, let me know, which GCTF version you are using under TomoBEAR

Could you let me know as well whether you can see SUCCESS files in the individual tomogram_XXX folders on the previous step?

P.S. I have separately freshly installed TomoBEAR-v0.3.0 and imod 4.11.24, but was not able to reproduce the error. However, inspecting the issue now I think it might not be related to IMOD, but to the failed CTF estimation on the previous step, since based on your logs both .defocus file itself and softlink to it on do exist, but the file itself is empty.

[wlugmayr]

I tried it with Gctf-v1.06_sm_30_cu8.0_x86_64 and GCTF_v1.18_sm30-75_cu10.1 and both produce a tomogram_001.defocus with content line '1 0 0. 0. 0 3'. Should the defocus file not contain more lines?

[wlugmayr]

Job 7_GCTFCtfphaseflipCTFCorrection_1/tomogram_001/output.json contains:

{"previous_step":"GCTFCtfphaseflipCTFCorrection", "previous_step_output_folder":"./output/7_GCTFCtfphaseflipCTFCorrection_1/tomogram_001", "defocus_slice_folder_path":"./output/7_GCTFCtfphaseflipCTFCorrection_1/tomogram_001/slices", "global_lower_defocus_average_in_angstrom":12266.627499999999, "global_upper_defocus_average_in_angstrom":36799.882499999992}%

and has a SUCCESS file

The output of GCTFCtfphaseflipCTFCorrection.log seems to be ok. ################################################################################################################################# All 58 files finished successfully:

TIME<<<<<< TOTAL: 100.281367s #################################################################################################################################

[ArtsemiY]

I tried it with Gctf-v1.06_sm_30_cu8.0_x86_64 and GCTF_v1.18_sm30-75_cu10.1 and both produce a tomogram_001.defocus with content line '1 0 0. 0. 0 3'. Should the defocus file not contain more lines?

Yes, .defocus file should have many lines of output - per each view, basically I have checked my setup for this case, I have also used GCTF_v1.18_sm30-75_cu10.1 while testing this recently

If that is possible, I would ask to provide me by e-mail ( artsemi.yushkevich@mdc-berlin.de ) with the archived folder containing the following:

• project directories output/ and scratch/
• exact copy of input configuration file ribosome_empiar_10064_dynamo.json - just in case smth was changed
• defaults configuration file configurations/defaults.json from your TomoBEAR installation so that I could inspect different steps to figure out the root of the issue.

[ArtsemiY]

Dear @wlugmayr

I have good news. After looking at your files I was finally able to reproduce your issue and find the possible reason.

Please, try to change the "processing_path" value to the absolute path in the "general" section of the ribosome_empiar_10064_dynamo.json. Before restarting TomoBEAR after changes you need to delete folders of steps 8_BatchRunTomo_1 and 7_GCTFCtfphaseflipCTFCorrection_1 from output.

P.S. TomoBEAR makes several other non-general assumptions on paths during post-installation initialization, pre-execution setup and during execution itself, but we will do our best to fix all those limitations in the future releases.

[wlugmayr]

Yes, using the absolute processing path in the json file solved this problem.

[ArtsemiY]

Cool, happy to hear that! Than I am closing this issue