multi-body refinement FFTW crash

[Pallesen-IUB]

This crash is persistent in versions 3.1 and 4.0.

data set: Two optics groups box sizes 400 and 632 pixel sizes 1.054 and 0.666 number of molecular projection images: ~30,000

Job: Multibody refinement with 3 mpi ranks; threads per rank 1 (--j 1) Running on 2 RTX8000 cards on HP Z8 linux tower; Ubuntu

Command: time mpiexec -np 3 which relion_refine_mpi --ref_angpix 1.054 --continue Refine3D/cls6c3mask_it018_optimiser.star --o MultiBody/twobody --solvent_correct_fsc --multibody_masks Bodies.star --oversampling 1 --healpix_order 3 --auto_local_healpix_order 4 --offset_range 3 --offset_step 1.5 --skip_gridding --reconstruct_subtracted_bodies --dont_combine_weights_via_disc --pool 30 --pad 2 --j 1 --gpu "" 1> ./mrefine.out 2> ./mrefine.err

error log (v3.1): No protocol specified

• WARNING: will ignore (but maintain) values for the unknown label: rlnSgdInitialIterations

• WARNING: will ignore (but maintain) values for the unknown label: rlnSgdFinalIterations

• WARNING: will ignore (but maintain) values for the unknown label: rlnSgdInBetweenIterations

• WARNING: will ignore (but maintain) values for the unknown label: rlnSgdInitialIterations

• WARNING: will ignore (but maintain) values for the unknown label: rlnSgdInitialIterations + WARNING: will ignore (but maintain) values for the unknown label: rlnSgdFinalIterations

• WARNING: will ignore (but maintain) values for the unknown label: rlnSgdInBetweenIterations

• WARNING: will ignore (but maintain) values for the unknown label: rlnSgdFinalIterations

• WARNING: will ignore (but maintain) values for the unknown label: rlnSgdInBetweenIterations The following warnings were encountered upon command-line parsing: WARNING: Option --ref_angpix is not a valid RELION argument fFourier= Size(Y,X): 632x317 i=[0..631] j=[0..316] inputFourier= Size(Y,X): 400x201 i=[0..399] j=[0..200] in: /root/relion4/relion/src/fftw.cpp, line 325 ERROR: BUG: incompatible shaped in setFourier part of FFTW transformer in: /root/relion4/relion/src/fftw.cpp, line 325 ERROR: ERROR: BUG: incompatible shaped in setFourier part of FFTW transformer follower 1 encountered error: === Backtrace === /root/relion4/relion/build/bin/relion_refine_mpi(_ZN11RelionErrorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_l+0x7d) [0x563eec1db87d] /root/relion4/relion/build/bin/relion_refine_mpi(+0x9da11) [0x563eec1bca11] /root/relion4/relion/build/bin/relion_refine_mpi(+0x2bce1d) [0x563eec3dbe1d] /lib/x86_64-linux-gnu/libgomp.so.1(GOMP_parallel+0x46) [0x7f3906e6b8e6] /root/relion4/relion/build/bin/relion_refine_mpi(_ZN11MlOptimiser24expectationSomeParticlesEll+0x1188) [0x563eec3cc148] /root/relion4/relion/build/bin/relion_refine_mpi(_ZN14MlOptimiserMpi11expectationEv+0x294d) [0x563eec20016d] /root/relion4/relion/build/bin/relion_refine_mpi(_ZN14MlOptimiserMpi7iterateEv+0xfb) [0x563eec20c9ab] /root/relion4/relion/build/bin/relion_refine_mpi(main+0x79) [0x563eec1c6cb9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f3906c730b3] /root/relion4/relion/build/bin/relion_refine_mpi(_start+0x2e) [0x563eec1c9fde]

  ERROR: BUG: incompatible shaped in setFourier part of FFTW transformer

  MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them.

equivalent autorefine job runs without any FFTW-related issues (obviously; I'm continuing this run).

[biochem-fan]

I'm not sure if MultiBody refinement supports optics groups with different box sizes. What happens if you re-extract particles in the second optics group into a 400 px box? It will be 632 * 0.666 / 400 = 1.052 A/px. Since the first optics group is 1.054 A/px, there is little point having the second group at a much smaller pixel size.

[Pallesen-IUB]

I kept them at different box sizes for other reasons. As you picked up easily, they are scaled properly... I thought about this, but read this thread: https://github.com/3dem/relion/issues/543 This user shared a star header revealing box sizes of 384 and 400 and with no apparent issues...

I'll hack, re-extract and re-run...

[Pallesen-IUB]

Runs as expected with same box size across optics groups.

[scheres]

thanks for confirming!