mpi_refine thread errors.

[yuanhaowang1213]

I tried with relion refine, but give report thread error. It is strange, because one day ago, I run the same code, it could work perfectly.

Environment:

• OS: Ubuntu 18.04
• MPI runtime: Openmpi 4.5
• RELION version RELION version: 3.1.2-commit-a9f327
• Memory: 256GB
• GPU: RTX TiTan

Job options:

• Type of job: [e.g. Refine3D]
• Number of MPI processes: 3
• Number of threads: 6
• Full command (see note.txt in the job directory):

  mpirun -np 3 `which relion_refine_mpi` --o Refine3D/run --auto_refine --split_random_halves --i particles_subtomo.star --ref riboReScale200.mrc --firstiter_cc --ini_high 60 --dont_combine_weights_via_disc --pool 30 --pad 2 --ctf --particle_diameter 350 --flatten_solvent --zero_mask --solvent_mask MaskCreate_loose4/mask.mrc --oversampling 1 --healpix_order 2 --auto_local_healpix_order 4 --offset_range 8 --offset_step 2 --sym C1 --low_resol_join_halves 40 --norm --scale  --gpu --j 6

Error message:

RELION version: 3.1.2-commit-a9f327 Precision: BASE=double, CUDA-ACC=single

=== RELION MPI setup ===

• Number of MPI processes = 3
• Number of threads per MPI process = 6
• Total number of threads therefore = 18
• Leader (0) runs on host = kw60609
• Follower 1 runs on host = kw60609

• Follower 2 runs on host = kw60609

  uniqueHost kw60609 has 2 ranks. GPU-ids not specified for this rank, threads will automatically be mapped to available devices. Thread 0 on follower 1 mapped to device 0 Thread 1 on follower 1 mapped to device 0 Thread 2 on follower 1 mapped to device 0 Thread 3 on follower 1 mapped to device 0 Thread 4 on follower 1 mapped to device 0 Thread 5 on follower 1 mapped to device 0 GPU-ids not specified for this rank, threads will automatically be mapped to available devices. Thread 0 on follower 2 mapped to device 0 Thread 1 on follower 2 mapped to device 0 Thread 2 on follower 2 mapped to device 0 Thread 3 on follower 2 mapped to device 0 Thread 4 on follower 2 mapped to device 0 Thread 5 on follower 2 mapped to device 0 Device 0 on kw60609 is split between 2 followers [kw60609:21009] Process received signal [kw60609:21009] Signal: Segmentation fault (11) [kw60609:21009] Signal code: Address not mapped (1) [kw60609:21009] Failing at address: (nil) [kw60609:21009] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x1457df5fc980] [kw60609:21009] [ 1] /home/wangy0k/Downloads/software/relion3.1/build/bin/relion_refine_mpi(_ZN10Experiment4readE8FileNamebbbbi+0xe4c)[0x55e4b25a872c] [kw60609:21009] [ 2] /home/wangy0k/Downloads/software/relion3.1/build/bin/relion_refine_mpi(_ZN11MlOptimiser17initialiseGeneralEi+0x38b)[0x55e4b26afa8b] [kw60609:21009] [ 3] /home/wangy0k/Downloads/software/relion3.1/build/bin/relion_refine_mpi(_ZN14MlOptimiserMpi10initialiseEv+0xa97)[0x55e4b2518797] [kw60609:21009] [ 4] /home/wangy0k/Downloads/software/relion3.1/build/bin/relion_refine_mpi(main+0x6b)[0x55e4b24dd4bb] [kw60609:21009] [ 5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x1457de2a1bf7] [kw60609:21009] [ 6] /home/wangy0k/Downloads/software/relion3.1/build/bin/relion_refine_mpi(_start+0x2a)[0x55e4b24e05da] [kw60609:21009] End of error message

  ==================

[biochem-fan]

It is strange, because one day ago, I run the same code, it could work perfectly.

If exactly the same command worked before, probably it is not RELION's problem. Didn't you move some files?

--o Refine3D/run

Unusual layout. Wh don't you have jobXXX folder?

[yuanhaowang1213]

It is strange, because one day ago, I run the same code, it could work perfectly.

If exactly the same command worked before, probably it is not RELION's problem. Didn't you move some files? Nope. No files are removed. Recompied Relion 3.1, still the same thing.

--o Refine3D/run

Unusual layout. Wh don't you have jobXXX folder? Because I write the command into a bash file, to do it automatically.

[yuanhaowang1213]

Sorry. It is the omp conflict. Fixed.

Thank you for your timing reply.

[tomgreen66]

Sorry. It is the omp conflict. Fixed.

Thank you for your timing reply.

We have a similar issue - could you explain what "omp conflict" refers too?