corrupted size vs. prev_size with relion_refine_mpi

[heejongkim]

This is a template for reporting bugs. Please fill in as much information as you can.

Suddenly, with both 3.1.2 and 3.1.3, this error started showing up with relion_refine_mpi jobs, including initial model, 3d classification and so forth.

Environment:

• OS: [e.g. Ubuntu 16.04 LTS] Centos7
• MPI runtime: [e.g. OpenMPI 2.0.1] MPICH 3.4.1
• RELION version [e.g. RELION-3.1-devel-commit-6ba935 (please see the title bar of the GUI)] 3.1.2 and 3.1.3 release version
• Memory: [e.g. 128 GB] 376GB
• GPU: [e.g. GTX 1080Ti] 2080Ti

Dataset:

• Box size: [e.g. 256 px] 400 px (bin2)
• Pixel size: [e.g. 0.9 Å/px] 0.86 (x2 for bin2)
• Number of particles: [e.g. 150,000] 200k
• Description: [e.g. A tetrameric protein of about 400 kDa in total] 2-300 kDa

Job options:

• Type of job: [e.g. Refine3D] Initial, 3D class
• Number of MPI processes: [e.g. 4] 11
• Number of threads: [e.g. 6] 6
• Full command (see note.txt in the job directory):

  `which relion_refine_mpi` --o InitialModel/job017/run --sgd_ini_iter 100 --sgd_inbetween_iter 200 --sgd_fin_iter 100 --sgd_write_iter 10 --sgd_ini_resol 60 --sgd_fin_resol 15 --sgd_ini_subset 100 --sgd_fin_subset 500 --sgd  --denovo_3dref --i Extract/job016/particles.star --ctf --K 3 --sym C1 --flatten_solvent  --zero_mask  --dont_combine_weights_via_disc --scratch_dir /ssd --pool 50 --pad 2  --particle_diameter 200 --oversampling 1 --healpix_order 2 --offset_range 6 --offset_step 2 --j 6 --gpu "" --keep_scratch --pipeline_control InitialModel/job017/

Error message:

Please cite the full error message as the example below.

*** Error in `/opt/relion-3.1.3/bin/relion_refine_mpi': corrupted size vs. prev_size: 0x00000000019846d0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x80d97)[0x7f8c2395bd97]
/lib64/libc.so.6(+0x8136e)[0x7f8c2395c36e]
/lib64/libnvidia-fatbinaryloader.so.430.26(+0x1e18a)[0x7f8c2083918a]
/lib64/libnvidia-fatbinaryloader.so.430.26(+0x1ba2c)[0x7f8c20836a2c]
/lib64/libnvidia-fatbinaryloader.so.430.26(+0x1170a)[0x7f8c2082c70a]
/lib64/libnvidia-fatbinaryloader.so.430.26(+0x1b2a1)[0x7f8c208362a1]
/lib64/libnvidia-fatbinaryloader.so.430.26(+0x27c3d)[0x7f8c20842c3d]
/lib64/libc.so.6(__cxa_finalize+0x9a)[0x7f8c23914eda]
/lib64/libnvidia-fatbinaryloader.so.430.26(+0x5cb6)[0x7f8c20820cb6]
======= Memory map: ========
00400000-00a24000 r-xp 00000000 08:23 25699894                           /opt/relion-3.1.3/bin/relion_refine_mpi
00c23000-00c24000 r--p 00623000 08:23 25699894                           /opt/relion-3.1.3/bin/relion_refine_mpi
00c24000-00c25000 rw-p 00624000 08:23 25699894                           /opt/relion-3.1.3/bin/relion_refine_mpi
00c25000-00c65000 rw-p 00000000 00:00 0 
017cb000-8d53e000 rw-p 00000000 00:00 0                                  [heap]
200000000-c00200000 ---p 00000000 00:00 0 
7f88c8000000-7f88c87d0000 rw-p 00000000 00:00 0 
7f88c87d0000-7f88cc000000 ---p 00000000 00:00 0 
7f8bb0000000-7f8bb0021000 rw-p 00000000 00:00 0 
7f8bb0021000-7f8bb4000000 ---p 00000000 00:00 0 
7f8bb4000000-7f8bb5fe1000 rw-p 00000000 00:00 0 
7f8bb5fe1000-7f8bb8000000 ---p 00000000 00:00 0 
7f8bb8000000-7f8bb8021000 rw-p 00000000 00:00 0 
7f8bb8021000-7f8bbc000000 ---p 00000000 00:00 0 
7f8bc0000000-7f8bc091d000 rw-p 00000000 00:00 0 
7f8bc091d000-7f8bc4000000 ---p 00000000 00:00 0 
7f8bc8000000-7f8bca38c000 rw-p 00000000 00:00 0 
7f8bca38c000-7f8bcc000000 ---p 00000000 00:00 0 
7f8bd0000000-7f8bd0a2d000 rw-p 00000000 00:00 0 
7f8bd0a2d000-7f8bd4000000 ---p 00000000 00:00 0 
7f8bd8000000-7f8bd8c0e000 rw-p 00000000 00:00 0 
7f8bd8c0e000-7f8bdc000000 ---p 00000000 00:00 0 
7f8bfd7fb000-7f8bfd7fc000 ---p 00000000 00:00 0 
7f8bfd7fc000-7f8bfdffc000 rw-p 00000000 00:00 0 
7f8bfdffc000-7f8bfdffd000 ---p 00000000 00:00 0 
7f8bfdffd000-7f8bfe7fd000 rw-p 00000000 00:00 0 
7f8c00000000-7f8c10000000 ---p 00000000 00:00 0 
7f8c13786000-7f8c1378e000 rw-s 00000000 00:13 306593                     /dev/shm/mpich_shar_tmpyBqGKe (deleted)
7f8c1378e000-7f8c1378f000 rw-s 00000000 00:13 306592                     /dev/shm/mpich_shar_tmp9czlFN (deleted)
7f8c13795000-7f8c13796000 ---p 00000000 00:00 0 
7f8c13796000-7f8c13fc0000 rw-p 00000000 00:00 0 
7f8c13fc0000-7f8c19fc0000 ---p 00000000 00:00 0 
7f8c19fc0000-7f8c19fd0000 -w-s 00000000 00:05 184432                     /dev/nvidia9
7f8c19fd0000-7f8c19fe0000 -w-s 00000000 00:05 180269                     /dev/nvidia8
7f8c19fe0000-7f8c19ff0000 -w-s 00000000 00:05 102016                     /dev/nvidia7
7f8c19ff0000-7f8c1a000000 -w-s 00000000 00:05 175290                     /dev/nvidia6
7f8c1a000000-7f8c1a010000 -w-s 00000000 00:05 170607                     /dev/nvidia5
7f8c1a010000-7f8c1a020000 -w-s 00000000 00:05 170507                     /dev/nvidia4
7f8c1a020000-7f8c1a030000 -w-s 00000000 00:05 167051                     /dev/nvidia3
7f8c1a030000-7f8c1a040000 -w-s 00000000 00:05 170403                     /dev/nvidia2
7f8c1a040000-7f8c1a050000 -w-s 00000000 00:05 159837                     /dev/nvidia1
7f8c1a050000-7f8c1a060000 -w-s 00000000 00:05 148719                     /dev/nvidia0
7f8c1a8b9000-7f8c1a8ba000 ---p 00000000 00:00 0 
7f8c1a8ba000-7f8c1b0ba000 rw-p 00000000 00:00 0 
7f8c1b0ba000-7f8c1b0bb000 ---p 00000000 00:00 0 
7f8c1b0bb000-7f8c1b8bb000 rw-p 00000000 00:00 0 
7f8c1b8bb000-7f8c1b90e000 r-xp 00000000 08:23 11289130                   /home/spuser/miniconda3/lib/libfabric/prov/librxm-fi.so
7f8c1b90e000-7f8c1bb0d000 ---p 00053000 08:23 11289130                   /home/spuser/miniconda3/lib/libfabric/prov/librxm-fi.so
7f8c1bb0d000-7f8c1bb11000 rw-p 00052000 08:23 11289130                   /home/spuser/miniconda3/lib/libfabric/prov/librxm-fi.so
7f8c1bb11000-7f8c1bb70000 r-xp 00000000 08:23 29780055                   /usr/lib64/libpsm2.so.2.1
7f8c1bb70000-7f8c1bd70000 ---p 0005f000 08:23 29780055                   /usr/lib64/libpsm2.so.2.1
7f8c1bd70000-7f8c1bd71000 r--p 0005f000 08:23 29780055                   /usr/lib64/libpsm2.so.2.1
7f8c1bd71000-7f8c1bd72000 rw-p 00060000 08:23 29780055                   /usr/lib64/libpsm2.so.2.1
7f8c1bd72000-7f8c1bd74000 rw-p 00000000 00:00 0 
7f8c1bd74000-7f8c1bdec000 r-xp 00000000 08:23 11289134                   /home/spuser/miniconda3/lib/libfabric/prov/libpsmx2-fi.so
7f8c1bdec000-7f8c1bfeb000 ---p 00078000 08:23 11289134                   /home/spuser/miniconda3/lib/libfabric/prov/libpsmx2-fi.so
7f8c1bfeb000-7f8c1bff1000 rw-p 00077000 08:23 11289134                   /home/spuser/miniconda3/lib/libfabric/prov/libpsmx2-fi.so
7f8c1bff1000-7f8c1bff5000 rw-p 00000000 00:00 0 
7f8c1bff5000-7f8c1c04f000 r-xp 00000000 08:23 11289132                   /home/spuser/miniconda3/lib/libfabric/prov/libsockets-fi.so
7f8c1c04f000-7f8c1c24f000 ---p 0005a000 08:23 11289132                   /home/spuser/miniconda3/lib/libfabric/prov/libsockets-fi.so
7f8c1c24f000-7f8c1c253000 rw-p 0005a000 08:23 11289132                   /home/spuser/miniconda3/lib/libfabric/prov/libsockets-fi.so
7f8c1c253000-7f8c1c296000 r-xp 00000000 08:23 11289128                   /home/spuser/miniconda3/lib/libfabric/prov/libtcp-fi.so
7f8c1c296000-7f8c1c496000 ---p 00043000 08:23 11289128                   /home/spuser/miniconda3/lib/libfabric/prov/libtcp-fi.so
7f8c1c496000-7f8c1c499000 rw-p 00043000 08:23 11289128                   /home/spuser/miniconda3/lib/libfabric/prov/libtcp-fi.so
7f8c1c499000-7f8c1c4a3000 r-xp 00000000 08:23 29760907                   /usr/lib64/libnuma.so.1
7f8c1c4a3000-7f8c1c6a3000 ---p 0000a000 08:23 29760907                   /usr/lib64/libnuma.so.1
7f8c1c6a3000-7f8c1c6a4000 r--p 0000a000 08:23 29760907                   /usr/lib64/libnuma.so.1
7f8c1c6a4000-7f8c1c6a5000 rw-p 0000b000 08:23 29760907                   /usr/lib64/libnuma.so.1
7f8c1c6a5000-7f8c1c7d5000 r-xp 00000000 08:23 11289135                   /home/spuser/miniconda3/lib/libfabric/prov/libpsm3-fi.so
7f8c1c7d5000-7f8c1c9d4000 ---p 00130000 08:23 11289135                   /home/spuser/miniconda3/lib/libfabric/prov/libpsm3-fi.so
7f8c1c9d4000-7f8c1c9db000 rw-p 0012f000 08:23 11289135                   /home/spuser/miniconda3/lib/libfabric/prov/libpsm3-fi.so
7f8c1c9db000-7f8c1ca01000 rw-p 00000000 00:00 0 
7f8c1ca01000-7f8c1ca47000 r-xp 00000000 08:23 11289113                   /home/spuser/miniconda3/lib/libfabric/libfabric.so.1
7f8c1ca47000-7f8c1cc47000 ---p 00046000 08:23 11289113                   /home/spuser/miniconda3/lib/libfabric/libfabric.so.1
7f8c1cc47000-7f8c1cc4a000 rw-p 00046000 08:23 11289113                   /home/spuser/miniconda3/lib/libfabric/libfabric.so.1
7f8c1cc4a000-7f8c1cc96000 r-xp 00000000 08:23 11289129                   /home/spuser/miniconda3/lib/libfabric/prov/libshm-fi.so
7f8c1cc96000-7f8c1ce95000 ---p 0004c000 08:23 11289129                   /home/spuser/miniconda3/lib/libfabric/prov/libshm-fi.so
7f8c1ce95000-7f8c1ce99000 rw-p 0004b000 08:23 11289129                   /home/spuser/miniconda3/lib/libfabric/prov/libshm-fi.so
7f8c1fd60000-7f8c1fd6f000 r-xp 00000000 08:23 29759940                   /usr/lib64/libbz2.so.1.0.6
7f8c1fd6f000-7f8c1ff6e000 ---p 0000f000 08:23 29759940                   /usr/lib64/libbz2.so.1.0.6
7f8c1ff6e000-7f8c1ff6f000 r--p 0000e000 08:23 29759940                   /usr/lib64/libbz2.so.1.0.6
7f8c1ff6f000-7f8c1ff70000 rw-p 0000f000 08:23 29759940                   /usr/lib64/libbz2.so.1.0.6
7f8c1ff70000-7f8c1ff87000 r-xp 00000000 08:23 29776643                   /usr/lib64/libelf-0.172.so
7f8c1ff87000-7f8c20186000 ---p 00017000 08:23 29776643                   /usr/lib64/libelf-0.172.so
7f8c20186000-7f8c20187000 r--p 00016000 08:23 29776643                   /usr/lib64/libelf-0.172.so
7f8c20187000-7f8c20188000 rw-p 00017000 08:23 29776643                   /usr/lib64/libelf-0.172.so
7f8c20188000-7f8c2018c000 r-xp 00000000 08:23 29759596                   /usr/lib64/libattr.so.1.1.0
7f8c2018c000-7f8c2038b000 ---p 00004000 08:23 29759596                   /usr/lib64/libattr.so.1.1.0
7f8c2038b000-7f8c2038c000 r--p 00003000 08:23 29759596                   /usr/lib64/libattr.so.1.1.0
7f8c2038c000-7f8c2038d000 rw-p 00004000 08:23 29759596                   /usr/lib64/libattr.so.1.1.0
7f8c2038d000-7f8c203ab000 r-xp 00000000 08:23 29759988                   /usr/lib64/libnl-3.so.200.23.0
7f8c203ab000-7f8c205ab000 ---p 0001e000 08:23 29759988                   /usr/lib64/libnl-3.so.200.23.0
7f8c205ab000-7f8c205ad000 r--p 0001e000 08:23 29759988                   /usr/lib64/libnl-3.so.200.23.0
7f8c205ad000-7f8c205ae000 rw-p 00020000 08:23 29759988                   /usr/lib64/libnl-3.so.200.23.0
7f8c205ae000-7f8c20612000 r-xp 00000000 08:23 29759996                   /usr/lib64/libnl-route-3.so.200.23.0
7f8c20612000-7f8c20811000 ---p 00064000 08:23 29759996                   /usr/lib64/libnl-route-3.so.200.23.0
7f8c20811000-7f8c20814000 r--p 00063000 08:23 29759996                   /usr/lib64/libnl-route-3.so.200.23.0
7f8c20814000-7f8c20819000 rw-p 00066000 08:23 29759996                   /usr/lib64/libnl-route-3.so.200.23.0
7f8c20819000-7f8c2081b000 rw-p 00000000 00:00 0 
7f8c2081b000-7f8c20862000 r-xp 00000000 08:23 29761878                   /usr/lib64/libnvidia-fatbinaryloader.so.430.26
7f8c20862000-7f8c20a62000 ---p 00047000 08:23 29761878                   /usr/lib64/libnvidia-fatbinaryloader.so.430.26
7f8c20a62000-7f8c20a64000 rw-p 00047000 08:23 29761878                   /usr/lib64/libnvidia-fatbinaryloader.so.430.26
7f8c20a64000-7f8c20a69000 rw-p 00000000 00:00 0 
7f8c20a69000-7f8c20a8e000 r-xp 00000000 08:23 29759932                   /usr/lib64/liblzma.so.5.2.2
7f8c20a8e000-7f8c20c8d000 ---p 00025000 08:23 29759932                   /usr/lib64/liblzma.so.5.2.2
7f8c20c8d000-7f8c20c8e000 r--p 00024000 08:23 29759932                   /usr/lib64/liblzma.so.5.2.2
7f8c20c8e000-7f8c20c8f000 rw-p 00025000 08:23 29759932                   /usr/lib64/liblzma.so.5.2.2
7f8c20c8f000-7f8c20cdb000 r-xp 00000000 08:23 29765135                   /usr/lib64/libdw-0.172.so
7f8c20cdb000-7f8c20edb000 ---p 0004c000 08:23 29765135                   /usr/lib64/libdw-0.172.so
7f8c20edb000-7f8c20edd000 r--p 0004c000 08:23 29765135                   /usr/lib64/libdw-0.172.so
7f8c20edd000-7f8c20ede000 rw-p 0004e000 08:23 29765135                   /usr/lib64/libdw-0.172.so
7f8c20ede000-7f8c20ee2000 r-xp 00000000 08:23 29759598                   /usr/lib64/libcap.so.2.22
7f8c20ee2000-7f8c210e1000 ---p 00004000 08:23 29759598                   /usr/lib64/libcap.so.2.22
7f8c210e1000-7f8c210e2000 r--p 00003000 08:23 29759598                   /usr/lib64/libcap.so.2.22
7f8c210e2000-7f8c210e3000 rw-p 00004000 08:23 29759598                   /usr/lib64/libcap.so.2.22
7f8c210e3000-7f8c210f8000 r-xp 00000000 08:23 29759457                   /usr/lib64/libz.so.1.2.7
7f8c210f8000-7f8c212f7000 ---p 00015000 08:23 29759457                   /usr/lib64/libz.so.1.2.7
7f8c212f7000-7f8c212f8000 r--p 00014000 08:23 29759457                   /usr/lib64/libz.so.1.2.7
7f8c212f8000-7f8c212f9000 rw-p 00015000 08:23 29759457                   /usr/lib64/libz.so.1.2.7
7f8c212f9000-7f8c2133c000 r-xp 00000000 08:23 29759926                   /usr/lib64/libjpeg.so.62.1.0
7f8c2133c000-7f8c2153c000 ---p 00043000 08:23 29759926                   /usr/lib64/libjpeg.so.62.1.0
7f8c2153c000-7f8c2153d000 r--p 00043000 08:23 29759926                   /usr/lib64/libjpeg.so.62.1.0
7f8c2153d000-7f8c2153e000 rw-p 00044000 08:23 29759926                   /usr/lib64/libjpeg.so.62.1.0
7f8c2153e000-7f8c2154e000 rw-p 00000000 00:00 0 
7f8c2154e000-7f8c21557000 r-xp 00000000 08:23 29768747                   /usr/lib64/libjbig.so.2.0
7f8c21557000-7f8c21756000 ---p 00009000 08:23 29768747                   /usr/lib64/libjbig.so.2.0
7f8c21756000-7f8c21757000 r--p 00008000 08:23 29768747                   /usr/lib64/libjbig.so.2.0
7f8c21757000-7f8c2175a000 rw-p 00009000 08:23 29768747                   /usr/lib64/libjbig.so.2.0
7f8c2175a000-7f8c21760000 r-xp 00000000 08:23 3162208                    /opt/gcc_7.5.0/lib64/libatomic.so.1.2.0
7f8c21760000-7f8c2195f000 ---p 00006000 08:23 3162208                    /opt/gcc_7.5.0/lib64/libatomic.so.1.2.0
7f8c2195f000-7f8c21960000 r--p 00005000 08:23 3162208                    /opt/gcc_7.5.0/lib64/libatomic.so.1.2.0
7f8c21960000-7f8c21961000 rw-p 00006000 08:23 3162208                    /opt/gcc_7.5.0/lib64/libatomic.so.1.2.0
7f8c21961000-7f8c21962000 rw-p 00000000 00:00 0 
7f8c21962000-7f8c219af000 r-xp 00000000 08:23 29762881                   /usr/lib64/libnl.so.1.1.4
7f8c219af000-7f8c21baf000 ---p 0004d000 08:23 29762881                   /usr/lib64/libnl.so.1.1.4
7f8c21baf000-7f8c21bb0000 r--p 0004d000 08:23 29762881                   /usr/lib64/libnl.so.1.1.4
7f8c21bb0000-7f8c21bb5000 rw-p 0004e000 08:23 29762881                   /usr/lib64/libnl.so.1.1.4
7f8c21bb5000-7f8c21bca000 r-xp 00000000 08:23 29766308                   /usr/lib64/libibverbs.so.1.1.17.2
7f8c21bca000-7f8c21dca000 ---p 00015000 08:23 29766308                   /usr/lib64/libibverbs.so.1.1.17.2
7f8c21dca000-7f8c21dcb000 r--p 00015000 08:23 29766308                   /usr/lib64/libibverbs.so.1.1.17.2
7f8c21dcb000-7f8c21dcc000 rw-p 00016000 08:23 29766308                   /usr/lib64/libibverbs.so.1.1.17.2
7f8c21dcc000-7f8c21de1000 r-xp 00000000 08:23 29766998                   /usr/lib64/librdmacm.so.1.1.17.2
7f8c21de1000-7f8c21fe0000 ---p 00015000 08:23 29766998                   /usr/lib64/librdmacm.so.1.1.17.2
7f8c21fe0000-7f8c21fe1000 r--p 00014000 08:23 29766998                   /usr/lib64/librdmacm.so.1.1.17.2
7f8c21fe1000-7f8c21fe2000 rw-p 00015000 08:23 29766998                   /usr/lib64/librdmacm.so.1.1.17.2
7f8c21fe2000-7f8c21fe3000 rw-p 00000000 00:00 0 
7f8c21fe3000-7f8c22dcb000 r-xp 00000000 08:23 29760290                   /usr/lib64/libcuda.so.430.26
7f8c22dcb000-7f8c22fcb000 ---p 00de8000 08:23 29760290                   /usr/lib64/libcuda.so.430.26
7f8c22fcb000-7f8c23143000 rw-p 00de8000 08:23 29760290                   /usr/lib64/libcuda.so.430.26
7f8c23143000-7f8c23153000 rw-p 00000000 00:00 0 
7f8c23153000-7f8c232b2000 r-xp 00000000 08:23 29759936                   /usr/lib64/libxml2.so.2.9.1
7f8c232b2000-7f8c234b1000 ---p 0015f000 08:23 29759936                   /usr/lib64/libxml2.so.2.9.1
7f8c234b1000-7f8c234b9000 r--p 0015e000 08:23 29759936                   /usr/lib64/libxml2.so.2.9.1
7f8c234b9000-7f8c234bb000 rw-p 00166000 08:23 29759936                   /usr/lib64/libxml2.so.2.9.1
7f8c234bb000-7f8c234bd000 rw-p 00000000 00:00 0 
7f8c234bd000-7f8c234d2000 r-xp 00000000 08:23 29765150                   /usr/lib64/libudev.so.1.6.2
7f8c234d2000-7f8c236d1000 ---p 00015000 08:23 29765150                   /usr/lib64/libudev.so.1.6.2
7f8c236d1000-7f8c236d2000 r--p 00014000 08:23 29765150                   /usr/lib64/libudev.so.1.6.2
7f8c236d2000-7f8c236d3000 rw-p 00015000 08:23 29765150                   /usr/lib64/libudev.so.1.6.2
7f8c236d3000-7f8c236da000 r-xp 00000000 08:23 29759195                   /usr/lib64/librt-2.17.so
7f8c236da000-7f8c238d9000 ---p 00007000 08:23 29759195                   /usr/lib64/librt-2.17.so
7f8c238d9000-7f8c238da000 r--p 00006000 08:23 29759195                   /usr/lib64/librt-2.17.so
7f8c238da000-7f8c238db000 rw-p 00007000 08:23 29759195                   /usr/lib64/librt-2.17.so
7f8c238db000-7f8c23a9d000 r-xp 00000000 08:23 29759165                   /usr/lib64/libc-2.17.so
7f8c23a9d000-7f8c23c9d000 ---p 001c2000 08:23 29759165                   /usr/lib64/libc-2.17.so
7f8c23c9d000-7f8c23ca1000 r--p 001c2000 08:23 29759165                   /usr/lib64/libc-2.17.so
7f8c23ca1000-7f8c23ca3000 rw-p 001c6000 08:23 29759165                   /usr/lib64/libc-2.17.so
7f8c23ca3000-7f8c23ca8000 rw-p 00000000 00:00 0 
7f8c23ca8000-7f8c23cbf000 r-xp 00000000 08:23 29759191                   /usr/lib64/libpthread-2.17.so
7f8c23cbf000-7f8c23ebe000 ---p 00017000 08:23 29759191                   /usr/lib64/libpthread-2.17.so
7f8c23ebe000-7f8c23ebf000 r--p 00016000 08:23 29759191                   /usr/lib64/libpthread-2.17.so
7f8c23ebf000-7f8c23ec0000 rw-p 00017000 08:23 29759191                   /usr/lib64/libpthread-2.17.so
7f8c23ec0000-7f8c23ec4000 rw-p 00000000 00:00 0 
7f8c23ec4000-7f8c23eda000 r-xp 00000000 08:23 3161869                    /opt/gcc_7.5.0/lib64/libgcc_s.so.1
7f8c23eda000-7f8c240d9000 ---p 00016000 08:23 3161869                    /opt/gcc_7.5.0/lib64/libgcc_s.so.1
7f8c240d9000-7f8c240da000 r--p 00015000 08:23 3161869                    /opt/gcc_7.5.0/lib64/libgcc_s.so.1
7f8c240da000-7f8c240db000 rw-p 00016000 08:23 3161869                    /opt/gcc_7.5.0/lib64/libgcc_s.so.1
7f8c240db000-7f8c24108000 r-xp 00000000 08:23 3162222                    /opt/gcc_7.5.0/lib64/libgomp.so.1.0.0
7f8c24108000-7f8c24307000 ---p 0002d000 08:23 3162222                    /opt/gcc_7.5.0/lib64/libgomp.so.1.0.0
7f8c24307000-7f8c24308000 r--p 0002c000 08:23 3162222                    /opt/gcc_7.5.0/lib64/libgomp.so.1.0.0
7f8c24308000-7f8c24309000 rw-p 0002d000 08:23 3162222                    /opt/gcc_7.5.0/lib64/libgomp.so.1.0.0
7f8c24309000-7f8c2440a000 r-xp 00000000 08:23 29759173                   /usr/lib64/libm-2.17.so
7f8c2440a000-7f8c24609000 ---p 00101000 08:23 29759173                   /usr/lib64/libm-2.17.so
7f8c24609000-7f8c2460a000 r--p 00100000 08:23 29759173                   /usr/lib64/libm-2.17.so
7f8c2460a000-7f8c2460b000 rw-p 00101000 08:23 29759173                   /usr/lib64/libm-2.17.so
7f8c2460b000-7f8c2477e000 r-xp 00000000 08:23 3162144                    /opt/gcc_7.5.0/lib64/libstdc++.so.6.0.24
7f8c2477e000-7f8c2497e000 ---p 00173000 08:23 3162144                    /opt/gcc_7.5.0/lib64/libstdc++.so.6.0.24
7f8c2497e000-7f8c24988000 r--p 00173000 08:23 3162144                    /opt/gcc_7.5.0/lib64/libstdc++.so.6.0.24
7f8c24988000-7f8c2498a000 rw-p 0017d000 08:23 3162144                    /opt/gcc_7.5.0/lib64/libstdc++.so.6.0.24
7f8c2498a000-7f8c2498e000 rw-p 00000000 00:00 0 
7f8c2498e000-7f8c24a07000 r-xp 00000000 08:23 2888908                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2.89
7f8c24a07000-7f8c24c06000 ---p 00079000 08:23 2888908                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2.89
7f8c24c06000-7f8c24c0a000 rw-p 00078000 08:23 2888908                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2.89
7f8c24c0a000-7f8c24c0c000 rw-p 00000000 00:00 0 
7f8c24c0c000-7f8c24c35000 r-xp 00000000 08:23 29759929                   /usr/lib64/libpng15.so.15.13.0
7f8c24c35000-7f8c24e35000 ---p 00029000 08:23 29759929                   /usr/lib64/libpng15.so.15.13.0
7f8c24e35000-7f8c24e36000 r--p 00029000 08:23 29759929                   /usr/lib64/libpng15.so.15.13.0
7f8c24e36000-7f8c24e37000 rw-p 0002a000 08:23 29759929                   /usr/lib64/libpng15.so.15.13.0
7f8c24e37000-7f8c273b5000 r-xp 00000000 08:23 2888877                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcurand.so.10.1.2.89
7f8c273b5000-7f8c275b4000 ---p 0257e000 08:23 2888877                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcurand.so.10.1.2.89
7f8c275b4000-7f8c28983000 rw-p 0257d000 08:23 2888877                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcurand.so.10.1.2.89
7f8c28983000-7f8c28eda000 rw-p 00000000 00:00 0 
7f8c28eda000-7f8c290e7000 r-xp 00000000 08:23 25699833                   /opt/relion-3.1.3/lib/libfftw3f.so.3.5.8
7f8c290e7000-7f8c292e6000 ---p 0020d000 08:23 25699833                   /opt/relion-3.1.3/lib/libfftw3f.so.3.5.8
7f8c292e6000-7f8c292f7000 r--p 0020c000 08:23 25699833                   /opt/relion-3.1.3/lib/libfftw3f.so.3.5.8
7f8c292f7000-7f8c292f8000 rw-p 0021d000 08:23 25699833                   /opt/relion-3.1.3/lib/libfftw3f.so.3.5.8
7f8c292f8000-7f8c2947d000 r-xp 00000000 08:23 25699835                   /opt/relion-3.1.3/lib/libfftw3.so.3.5.8
7f8c2947d000-7f8c2967c000 ---p 00185000 08:23 25699835                   /opt/relion-3.1.3/lib/libfftw3.so.3.5.8
7f8c2967c000-7f8c29688000 r--p 00184000 08:23 25699835                   /opt/relion-3.1.3/lib/libfftw3.so.3.5.8
7f8c29688000-7f8c29689000 rw-p 00190000 08:23 25699835                   /opt/relion-3.1.3/lib/libfftw3.so.3.5.8
7f8c29689000-7f8c296f8000 r-xp 00000000 08:23 29768750                   /usr/lib64/libtiff.so.5.2.0
7f8c296f8000-7f8c298f8000 ---p 0006f000 08:23 29768750                   /usr/lib64/libtiff.so.5.2.0
7f8c298f8000-7f8c298f9000 r--p 0006f000 08:23 29768750                   /usr/lib64/libtiff.so.5.2.0
7f8c298f9000-7f8c298fc000 rw-p 00070000 08:23 29768750                   /usr/lib64/libtiff.so.5.2.0
7f8c298fc000-7f8c298fd000 rw-p 00000000 00:00 0 
7f8c298fd000-7f8c298ff000 r-xp 00000000 08:23 29759171                   /usr/lib64/libdl-2.17.so
7f8c298ff000-7f8c29aff000 ---p 00002000 08:23 29759171                   /usr/lib64/libdl-2.17.so
7f8c29aff000-7f8c29b00000 r--p 00002000 08:23 29759171                   /usr/lib64/libdl-2.17.so
7f8c29b00000-7f8c29b01000 rw-p 00003000 08:23 29759171                   /usr/lib64/libdl-2.17.so
7f8c29b01000-7f8c2a7d4000 r-xp 00000000 08:23 5523962                    /opt/mpich-3.4.1/lib/libmpi.so.12.1.10
7f8c2a7d4000-7f8c2a9d4000 ---p 00cd3000 08:23 5523962                    /opt/mpich-3.4.1/lib/libmpi.so.12.1.10
7f8c2a9d4000-7f8c2a9fb000 r--p 00cd3000 08:23 5523962                    /opt/mpich-3.4.1/lib/libmpi.so.12.1.10
7f8c2a9fb000-7f8c2aa08000 rw-p 00cfa000 08:23 5523962                    /opt/mpich-3.4.1/lib/libmpi.so.12.1.10
7f8c2aa08000-7f8c2aa67000 rw-p 00000000 00:00 0 
7f8c2aa67000-7f8c2aa85000 r-xp 00000000 08:23 5523970                    /opt/mpich-3.4.1/lib/libmpicxx.so.12.1.10
7f8c2aa85000-7f8c2ac85000 ---p 0001e000 08:23 5523970                    /opt/mpich-3.4.1/lib/libmpicxx.so.12.1.10
7f8c2ac85000-7f8c2ac87000 r--p 0001e000 08:23 5523970                    /opt/mpich-3.4.1/lib/libmpicxx.so.12.1.10
7f8c2ac87000-7f8c2ac88000 rw-p 00020000 08:23 5523970                    /opt/mpich-3.4.1/lib/libmpicxx.so.12.1.10
7f8c2ac88000-7f8c33e9e000 r-xp 00000000 08:23 2888874                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcufft.so.10.1.2.89
7f8c33e9e000-7f8c3409d000 ---p 09216000 08:23 2888874                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcufft.so.10.1.2.89
7f8c3409d000-7f8c340af000 rw-p 09215000 08:23 2888874                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcufft.so.10.1.2.89
7f8c340af000-7f8c34128000 rw-p 00000000 00:00 0 
7f8c34128000-7f8c3414a000 r-xp 00000000 08:23 29758944                   /usr/lib64/ld-2.17.so
7f8c34301000-7f8c34316000 rw-p 00000000 00:00 0 
7f8c34346000-7f8c34349000 rw-p 00000000 00:00 0 
7f8c34349000-7f8c3434a000 r--p 00021000 08:23 29758944                   /usr/lib64/ld-2.17.so
7f8c3434a000-7f8c3434b000 rw-p 00022000 08:23 29758944                   /usr/lib64/ld-2.17.so
7f8c3434b000-7f8c3434c000 rw-p 00000000 00:00 0 
7fff84b64000-7fff84b86000 rw-p 00000000 00:00 0                          [stack]
7fff84bd4000-7fff84bd6000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
*** Error in `/opt/relion-3.1.3/bin/relion_refine_mpi': corrupted size vs. prev_size: 0x0000000002abf6d0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x80d97)[0x7fb75f75bd97]
/lib64/libc.so.6(+0x8136e)[0x7fb75f75c36e]
/lib64/libcuda.so.1(+0x208648)[0x7fb75dfeb648]
/lib64/libcuda.so.1(+0x20875d)[0x7fb75dfeb75d]
/lib64/libcuda.so.1(+0xf01ef)[0x7fb75ded31ef]
======= Memory map: ========
00400000-00a24000 r-xp 00000000 08:23 25699894                           /opt/relion-3.1.3/bin/relion_refine_mpi
00c23000-00c24000 r--p 00623000 08:23 25699894                           /opt/relion-3.1.3/bin/relion_refine_mpi
00c24000-00c25000 rw-p 00624000 08:23 25699894                           /opt/relion-3.1.3/bin/relion_refine_mpi
00c25000-00c65000 rw-p 00000000 00:00 0 
02906000-31256000 rw-p 00000000 00:00 0                                  [heap]
200000000-c00200000 ---p 00000000 00:00 0 
7fb404000000-7fb406a98000 rw-p 00000000 00:00 0 
7fb406a98000-7fb408000000 ---p 00000000 00:00 0 
7fb6ec000000-7fb6ec021000 rw-p 00000000 00:00 0 
7fb6ec021000-7fb6f0000000 ---p 00000000 00:00 0 
7fb6f0000000-7fb6f0a56000 rw-p 00000000 00:00 0 
7fb6f0a56000-7fb6f4000000 ---p 00000000 00:00 0 
7fb6f4000000-7fb6f4021000 rw-p 00000000 00:00 0 
7fb6f4021000-7fb6f8000000 ---p 00000000 00:00 0 
7fb6fc000000-7fb6fca42000 rw-p 00000000 00:00 0 
7fb6fca42000-7fb700000000 ---p 00000000 00:00 0 
7fb704000000-7fb7047cd000 rw-p 00000000 00:00 0 
7fb7047cd000-7fb708000000 ---p 00000000 00:00 0 
7fb70c000000-7fb70c9ae000 rw-p 00000000 00:00 0 
7fb70c9ae000-7fb710000000 ---p 00000000 00:00 0 
7fb714000000-7fb715229000 rw-p 00000000 00:00 0 
7fb715229000-7fb718000000 ---p 00000000 00:00 0 
7fb730000000-7fb740000000 ---p 00000000 00:00 0 
7fb7433c6000-7fb7433c7000 ---p 00000000 00:00 0 
7fb7433c7000-7fb743bc7000 rw-p 00000000 00:00 0 
7fb743bc7000-7fb743bc8000 ---p 00000000 00:00 0 
7fb743bc8000-7fb7443c8000 rw-p 00000000 00:00 0 
7fb74f586000-7fb74f58e000 rw-s 00000000 00:13 306593                     /dev/shm/mpich_shar_tmpyBqGKe (deleted)
7fb74f58e000-7fb74f58f000 rw-s 00000000 00:13 306592                     /dev/shm/mpich_shar_tmp9czlFN (deleted)
7fb74f595000-7fb74f596000 ---p 00000000 00:00 0 
7fb74f596000-7fb74fdc0000 rw-p 00000000 00:00 0 
7fb74fdc0000-7fb755dc0000 ---p 00000000 00:00 0 
7fb755dc0000-7fb755dd0000 -w-s 00000000 00:05 184432                     /dev/nvidia9
7fb755dd0000-7fb755de0000 -w-s 00000000 00:05 180269                     /dev/nvidia8
7fb755de0000-7fb755df0000 -w-s 00000000 00:05 102016                     /dev/nvidia7
7fb755df0000-7fb755e00000 -w-s 00000000 00:05 175290                     /dev/nvidia6
7fb755e00000-7fb755e10000 -w-s 00000000 00:05 170607                     /dev/nvidia5
7fb755e10000-7fb755e20000 -w-s 00000000 00:05 170507                     /dev/nvidia4
7fb755e20000-7fb755e30000 -w-s 00000000 00:05 167051                     /dev/nvidia3
7fb755e30000-7fb755e40000 -w-s 00000000 00:05 170403                     /dev/nvidia2
7fb755e40000-7fb755e50000 -w-s 00000000 00:05 159837                     /dev/nvidia1
7fb755e50000-7fb755e60000 -w-s 00000000 00:05 148719                     /dev/nvidia0
7fb7566b9000-7fb7566ba000 ---p 00000000 00:00 0 
7fb7566ba000-7fb756eba000 rw-p 00000000 00:00 0 
7fb756eba000-7fb756ebb000 ---p 00000000 00:00 0 
7fb756ebb000-7fb7576bb000 rw-p 00000000 00:00 0 
7fb7576bb000-7fb75770e000 r-xp 00000000 08:23 11289130                   /home/spuser/miniconda3/lib/libfabric/prov/librxm-fi.so
7fb75770e000-7fb75790d000 ---p 00053000 08:23 11289130                   /home/spuser/miniconda3/lib/libfabric/prov/librxm-fi.so
7fb75790d000-7fb757911000 rw-p 00052000 08:23 11289130                   /home/spuser/miniconda3/lib/libfabric/prov/librxm-fi.so
7fb757911000-7fb757970000 r-xp 00000000 08:23 29780055                   /usr/lib64/libpsm2.so.2.1
7fb757970000-7fb757b70000 ---p 0005f000 08:23 29780055                   /usr/lib64/libpsm2.so.2.1
7fb757b70000-7fb757b71000 r--p 0005f000 08:23 29780055                   /usr/lib64/libpsm2.so.2.1
7fb757b71000-7fb757b72000 rw-p 00060000 08:23 29780055                   /usr/lib64/libpsm2.so.2.1
7fb757b72000-7fb757b74000 rw-p 00000000 00:00 0 
7fb757b74000-7fb757bec000 r-xp 00000000 08:23 11289134                   /home/spuser/miniconda3/lib/libfabric/prov/libpsmx2-fi.so
7fb757bec000-7fb757deb000 ---p 00078000 08:23 11289134                   /home/spuser/miniconda3/lib/libfabric/prov/libpsmx2-fi.so
7fb757deb000-7fb757df1000 rw-p 00077000 08:23 11289134                   /home/spuser/miniconda3/lib/libfabric/prov/libpsmx2-fi.so
7fb757df1000-7fb757df5000 rw-p 00000000 00:00 0 
7fb757df5000-7fb757e4f000 r-xp 00000000 08:23 11289132                   /home/spuser/miniconda3/lib/libfabric/prov/libsockets-fi.so
7fb757e4f000-7fb75804f000 ---p 0005a000 08:23 11289132                   /home/spuser/miniconda3/lib/libfabric/prov/libsockets-fi.so
7fb75804f000-7fb758053000 rw-p 0005a000 08:23 11289132                   /home/spuser/miniconda3/lib/libfabric/prov/libsockets-fi.so
7fb758053000-7fb758096000 r-xp 00000000 08:23 11289128                   /home/spuser/miniconda3/lib/libfabric/prov/libtcp-fi.so
7fb758096000-7fb758296000 ---p 00043000 08:23 11289128                   /home/spuser/miniconda3/lib/libfabric/prov/libtcp-fi.so
7fb758296000-7fb758299000 rw-p 00043000 08:23 11289128                   /home/spuser/miniconda3/lib/libfabric/prov/libtcp-fi.so
7fb758299000-7fb7582a3000 r-xp 00000000 08:23 29760907                   /usr/lib64/libnuma.so.1
7fb7582a3000-7fb7584a3000 ---p 0000a000 08:23 29760907                   /usr/lib64/libnuma.so.1
7fb7584a3000-7fb7584a4000 r--p 0000a000 08:23 29760907                   /usr/lib64/libnuma.so.1
7fb7584a4000-7fb7584a5000 rw-p 0000b000 08:23 29760907                   /usr/lib64/libnuma.so.1
7fb7584a5000-7fb7585d5000 r-xp 00000000 08:23 11289135                   /home/spuser/miniconda3/lib/libfabric/prov/libpsm3-fi.so
7fb7585d5000-7fb7587d4000 ---p 00130000 08:23 11289135                   /home/spuser/miniconda3/lib/libfabric/prov/libpsm3-fi.so
7fb7587d4000-7fb7587db000 rw-p 0012f000 08:23 11289135                   /home/spuser/miniconda3/lib/libfabric/prov/libpsm3-fi.so
7fb7587db000-7fb758801000 rw-p 00000000 00:00 0 
7fb758801000-7fb758847000 r-xp 00000000 08:23 11289113                   /home/spuser/miniconda3/lib/libfabric/libfabric.so.1
7fb758847000-7fb758a47000 ---p 00046000 08:23 11289113                   /home/spuser/miniconda3/lib/libfabric/libfabric.so.1
7fb758a47000-7fb758a4a000 rw-p 00046000 08:23 11289113                   /home/spuser/miniconda3/lib/libfabric/libfabric.so.1
7fb758a4a000-7fb758a96000 r-xp 00000000 08:23 11289129                   /home/spuser/miniconda3/lib/libfabric/prov/libshm-fi.so
7fb758a96000-7fb758c95000 ---p 0004c000 08:23 11289129                   /home/spuser/miniconda3/lib/libfabric/prov/libshm-fi.so
7fb758c95000-7fb758c99000 rw-p 0004b000 08:23 11289129                   /home/spuser/miniconda3/lib/libfabric/prov/libshm-fi.so
7fb75bb60000-7fb75bb6f000 r-xp 00000000 08:23 29759940                   /usr/lib64/libbz2.so.1.0.6
7fb75bb6f000-7fb75bd6e000 ---p 0000f000 08:23 29759940                   /usr/lib64/libbz2.so.1.0.6
7fb75bd6e000-7fb75bd6f000 r--p 0000e000 08:23 29759940                   /usr/lib64/libbz2.so.1.0.6
7fb75bd6f000-7fb75bd70000 rw-p 0000f000 08:23 29759940                   /usr/lib64/libbz2.so.1.0.6
7fb75bd70000-7fb75bd87000 r-xp 00000000 08:23 29776643                   /usr/lib64/libelf-0.172.so
7fb75bd87000-7fb75bf86000 ---p 00017000 08:23 29776643                   /usr/lib64/libelf-0.172.so
7fb75bf86000-7fb75bf87000 r--p 00016000 08:23 29776643                   /usr/lib64/libelf-0.172.so
7fb75bf87000-7fb75bf88000 rw-p 00017000 08:23 29776643                   /usr/lib64/libelf-0.172.so
7fb75bf88000-7fb75bf8c000 r-xp 00000000 08:23 29759596                   /usr/lib64/libattr.so.1.1.0
7fb75bf8c000-7fb75c18b000 ---p 00004000 08:23 29759596                   /usr/lib64/libattr.so.1.1.0
7fb75c18b000-7fb75c18c000 r--p 00003000 08:23 29759596                   /usr/lib64/libattr.so.1.1.0
7fb75c18c000-7fb75c18d000 rw-p 00004000 08:23 29759596                   /usr/lib64/libattr.so.1.1.0
7fb75c18d000-7fb75c1ab000 r-xp 00000000 08:23 29759988                   /usr/lib64/libnl-3.so.200.23.0
7fb75c1ab000-7fb75c3ab000 ---p 0001e000 08:23 29759988                   /usr/lib64/libnl-3.so.200.23.0
7fb75c3ab000-7fb75c3ad000 r--p 0001e000 08:23 29759988                   /usr/lib64/libnl-3.so.200.23.0
7fb75c3ad000-7fb75c3ae000 rw-p 00020000 08:23 29759988                   /usr/lib64/libnl-3.so.200.23.0
7fb75c3ae000-7fb75c412000 r-xp 00000000 08:23 29759996                   /usr/lib64/libnl-route-3.so.200.23.0
7fb75c412000-7fb75c611000 ---p 00064000 08:23 29759996                   /usr/lib64/libnl-route-3.so.200.23.0
7fb75c611000-7fb75c614000 r--p 00063000 08:23 29759996                   /usr/lib64/libnl-route-3.so.200.23.0
7fb75c614000-7fb75c619000 rw-p 00066000 08:23 29759996                   /usr/lib64/libnl-route-3.so.200.23.0
7fb75c619000-7fb75c61b000 rw-p 00000000 00:00 0 
7fb75c61b000-7fb75c662000 r-xp 00000000 08:23 29761878                   /usr/lib64/libnvidia-fatbinaryloader.so.430.26
7fb75c662000-7fb75c862000 ---p 00047000 08:23 29761878                   /usr/lib64/libnvidia-fatbinaryloader.so.430.26
7fb75c862000-7fb75c864000 rw-p 00047000 08:23 29761878                   /usr/lib64/libnvidia-fatbinaryloader.so.430.26
7fb75c864000-7fb75c869000 rw-p 00000000 00:00 0 
7fb75c869000-7fb75c88e000 r-xp 00000000 08:23 29759932                   /usr/lib64/liblzma.so.5.2.2
7fb75c88e000-7fb75ca8d000 ---p 00025000 08:23 29759932                   /usr/lib64/liblzma.so.5.2.2
7fb75ca8d000-7fb75ca8e000 r--p 00024000 08:23 29759932                   /usr/lib64/liblzma.so.5.2.2
7fb75ca8e000-7fb75ca8f000 rw-p 00025000 08:23 29759932                   /usr/lib64/liblzma.so.5.2.2
7fb75ca8f000-7fb75cadb000 r-xp 00000000 08:23 29765135                   /usr/lib64/libdw-0.172.so
7fb75cadb000-7fb75ccdb000 ---p 0004c000 08:23 29765135                   /usr/lib64/libdw-0.172.so
7fb75ccdb000-7fb75ccdd000 r--p 0004c000 08:23 29765135                   /usr/lib64/libdw-0.172.so
7fb75ccdd000-7fb75ccde000 rw-p 0004e000 08:23 29765135                   /usr/lib64/libdw-0.172.so
7fb75ccde000-7fb75cce2000 r-xp 00000000 08:23 29759598                   /usr/lib64/libcap.so.2.22
7fb75cce2000-7fb75cee1000 ---p 00004000 08:23 29759598                   /usr/lib64/libcap.so.2.22
7fb75cee1000-7fb75cee2000 r--p 00003000 08:23 29759598                   /usr/lib64/libcap.so.2.22
7fb75cee2000-7fb75cee3000 rw-p 00004000 08:23 29759598                   /usr/lib64/libcap.so.2.22
7fb75cee3000-7fb75cef8000 r-xp 00000000 08:23 29759457                   /usr/lib64/libz.so.1.2.7
7fb75cef8000-7fb75d0f7000 ---p 00015000 08:23 29759457                   /usr/lib64/libz.so.1.2.7
7fb75d0f7000-7fb75d0f8000 r--p 00014000 08:23 29759457                   /usr/lib64/libz.so.1.2.7
7fb75d0f8000-7fb75d0f9000 rw-p 00015000 08:23 29759457                   /usr/lib64/libz.so.1.2.7
7fb75d0f9000-7fb75d13c000 r-xp 00000000 08:23 29759926                   /usr/lib64/libjpeg.so.62.1.0
7fb75d13c000-7fb75d33c000 ---p 00043000 08:23 29759926                   /usr/lib64/libjpeg.so.62.1.0
7fb75d33c000-7fb75d33d000 r--p 00043000 08:23 29759926                   /usr/lib64/libjpeg.so.62.1.0
7fb75d33d000-7fb75d33e000 rw-p 00044000 08:23 29759926                   /usr/lib64/libjpeg.so.62.1.0
7fb75d33e000-7fb75d34e000 rw-p 00000000 00:00 0 
7fb75d34e000-7fb75d357000 r-xp 00000000 08:23 29768747                   /usr/lib64/libjbig.so.2.0
7fb75d357000-7fb75d556000 ---p 00009000 08:23 29768747                   /usr/lib64/libjbig.so.2.0
7fb75d556000-7fb75d557000 r--p 00008000 08:23 29768747                   /usr/lib64/libjbig.so.2.0
7fb75d557000-7fb75d55a000 rw-p 00009000 08:23 29768747                   /usr/lib64/libjbig.so.2.0
7fb75d55a000-7fb75d560000 r-xp 00000000 08:23 3162208                    /opt/gcc_7.5.0/lib64/libatomic.so.1.2.0
7fb75d560000-7fb75d75f000 ---p 00006000 08:23 3162208                    /opt/gcc_7.5.0/lib64/libatomic.so.1.2.0
7fb75d75f000-7fb75d760000 r--p 00005000 08:23 3162208                    /opt/gcc_7.5.0/lib64/libatomic.so.1.2.0
7fb75d760000-7fb75d761000 rw-p 00006000 08:23 3162208                    /opt/gcc_7.5.0/lib64/libatomic.so.1.2.0
7fb75d761000-7fb75d762000 rw-p 00000000 00:00 0 
7fb75d762000-7fb75d7af000 r-xp 00000000 08:23 29762881                   /usr/lib64/libnl.so.1.1.4
7fb75d7af000-7fb75d9af000 ---p 0004d000 08:23 29762881                   /usr/lib64/libnl.so.1.1.4
7fb75d9af000-7fb75d9b0000 r--p 0004d000 08:23 29762881                   /usr/lib64/libnl.so.1.1.4
7fb75d9b0000-7fb75d9b5000 rw-p 0004e000 08:23 29762881                   /usr/lib64/libnl.so.1.1.4
7fb75d9b5000-7fb75d9ca000 r-xp 00000000 08:23 29766308                   /usr/lib64/libibverbs.so.1.1.17.2
7fb75d9ca000-7fb75dbca000 ---p 00015000 08:23 29766308                   /usr/lib64/libibverbs.so.1.1.17.2
7fb75dbca000-7fb75dbcb000 r--p 00015000 08:23 29766308                   /usr/lib64/libibverbs.so.1.1.17.2
7fb75dbcb000-7fb75dbcc000 rw-p 00016000 08:23 29766308                   /usr/lib64/libibverbs.so.1.1.17.2
7fb75dbcc000-7fb75dbe1000 r-xp 00000000 08:23 29766998                   /usr/lib64/librdmacm.so.1.1.17.2
7fb75dbe1000-7fb75dde0000 ---p 00015000 08:23 29766998                   /usr/lib64/librdmacm.so.1.1.17.2
7fb75dde0000-7fb75dde1000 r--p 00014000 08:23 29766998                   /usr/lib64/librdmacm.so.1.1.17.2
7fb75dde1000-7fb75dde2000 rw-p 00015000 08:23 29766998                   /usr/lib64/librdmacm.so.1.1.17.2
7fb75dde2000-7fb75dde3000 rw-p 00000000 00:00 0 
7fb75dde3000-7fb75ebcb000 r-xp 00000000 08:23 29760290                   /usr/lib64/libcuda.so.430.26
7fb75ebcb000-7fb75edcb000 ---p 00de8000 08:23 29760290                   /usr/lib64/libcuda.so.430.26
7fb75edcb000-7fb75ef43000 rw-p 00de8000 08:23 29760290                   /usr/lib64/libcuda.so.430.26
7fb75ef43000-7fb75ef53000 rw-p 00000000 00:00 0 
7fb75ef53000-7fb75f0b2000 r-xp 00000000 08:23 29759936                   /usr/lib64/libxml2.so.2.9.1
7fb75f0b2000-7fb75f2b1000 ---p 0015f000 08:23 29759936                   /usr/lib64/libxml2.so.2.9.1
7fb75f2b1000-7fb75f2b9000 r--p 0015e000 08:23 29759936                   /usr/lib64/libxml2.so.2.9.1
7fb75f2b9000-7fb75f2bb000 rw-p 00166000 08:23 29759936                   /usr/lib64/libxml2.so.2.9.1
7fb75f2bb000-7fb75f2bd000 rw-p 00000000 00:00 0 
7fb75f2bd000-7fb75f2d2000 r-xp 00000000 08:23 29765150                   /usr/lib64/libudev.so.1.6.2
7fb75f2d2000-7fb75f4d1000 ---p 00015000 08:23 29765150                   /usr/lib64/libudev.so.1.6.2
7fb75f4d1000-7fb75f4d2000 r--p 00014000 08:23 29765150                   /usr/lib64/libudev.so.1.6.2
7fb75f4d2000-7fb75f4d3000 rw-p 00015000 08:23 29765150                   /usr/lib64/libudev.so.1.6.2
7fb75f4d3000-7fb75f4da000 r-xp 00000000 08:23 29759195                   /usr/lib64/librt-2.17.so
7fb75f4da000-7fb75f6d9000 ---p 00007000 08:23 29759195                   /usr/lib64/librt-2.17.so
7fb75f6d9000-7fb75f6da000 r--p 00006000 08:23 29759195                   /usr/lib64/librt-2.17.so
7fb75f6da000-7fb75f6db000 rw-p 00007000 08:23 29759195                   /usr/lib64/librt-2.17.so
7fb75f6db000-7fb75f89d000 r-xp 00000000 08:23 29759165                   /usr/lib64/libc-2.17.so
7fb75f89d000-7fb75fa9d000 ---p 001c2000 08:23 29759165                   /usr/lib64/libc-2.17.so
7fb75fa9d000-7fb75faa1000 r--p 001c2000 08:23 29759165                   /usr/lib64/libc-2.17.so
7fb75faa1000-7fb75faa3000 rw-p 001c6000 08:23 29759165                   /usr/lib64/libc-2.17.so
7fb75faa3000-7fb75faa8000 rw-p 00000000 00:00 0 
7fb75faa8000-7fb75fabf000 r-xp 00000000 08:23 29759191                   /usr/lib64/libpthread-2.17.so
7fb75fabf000-7fb75fcbe000 ---p 00017000 08:23 29759191                   /usr/lib64/libpthread-2.17.so
7fb75fcbe000-7fb75fcbf000 r--p 00016000 08:23 29759191                   /usr/lib64/libpthread-2.17.so
7fb75fcbf000-7fb75fcc0000 rw-p 00017000 08:23 29759191                   /usr/lib64/libpthread-2.17.so
7fb75fcc0000-7fb75fcc4000 rw-p 00000000 00:00 0 
7fb75fcc4000-7fb75fcda000 r-xp 00000000 08:23 3161869                    /opt/gcc_7.5.0/lib64/libgcc_s.so.1
7fb75fcda000-7fb75fed9000 ---p 00016000 08:23 3161869                    /opt/gcc_7.5.0/lib64/libgcc_s.so.1
7fb75fed9000-7fb75feda000 r--p 00015000 08:23 3161869                    /opt/gcc_7.5.0/lib64/libgcc_s.so.1
7fb75feda000-7fb75fedb000 rw-p 00016000 08:23 3161869                    /opt/gcc_7.5.0/lib64/libgcc_s.so.1
7fb75fedb000-7fb75ff08000 r-xp 00000000 08:23 3162222                    /opt/gcc_7.5.0/lib64/libgomp.so.1.0.0
7fb75ff08000-7fb760107000 ---p 0002d000 08:23 3162222                    /opt/gcc_7.5.0/lib64/libgomp.so.1.0.0
7fb760107000-7fb760108000 r--p 0002c000 08:23 3162222                    /opt/gcc_7.5.0/lib64/libgomp.so.1.0.0
7fb760108000-7fb760109000 rw-p 0002d000 08:23 3162222                    /opt/gcc_7.5.0/lib64/libgomp.so.1.0.0
7fb760109000-7fb76020a000 r-xp 00000000 08:23 29759173                   /usr/lib64/libm-2.17.so
7fb76020a000-7fb760409000 ---p 00101000 08:23 29759173                   /usr/lib64/libm-2.17.so
7fb760409000-7fb76040a000 r--p 00100000 08:23 29759173                   /usr/lib64/libm-2.17.so
7fb76040a000-7fb76040b000 rw-p 00101000 08:23 29759173                   /usr/lib64/libm-2.17.so
7fb76040b000-7fb76057e000 r-xp 00000000 08:23 3162144                    /opt/gcc_7.5.0/lib64/libstdc++.so.6.0.24
7fb76057e000-7fb76077e000 ---p 00173000 08:23 3162144                    /opt/gcc_7.5.0/lib64/libstdc++.so.6.0.24
7fb76077e000-7fb760788000 r--p 00173000 08:23 3162144                    /opt/gcc_7.5.0/lib64/libstdc++.so.6.0.24
7fb760788000-7fb76078a000 rw-p 0017d000 08:23 3162144                    /opt/gcc_7.5.0/lib64/libstdc++.so.6.0.24
7fb76078a000-7fb76078e000 rw-p 00000000 00:00 0 
7fb76078e000-7fb760807000 r-xp 00000000 08:23 2888908                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2.89
7fb760807000-7fb760a06000 ---p 00079000 08:23 2888908                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2.89
7fb760a06000-7fb760a0a000 rw-p 00078000 08:23 2888908                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2.89
7fb760a0a000-7fb760a0c000 rw-p 00000000 00:00 0 
7fb760a0c000-7fb760a35000 r-xp 00000000 08:23 29759929                   /usr/lib64/libpng15.so.15.13.0
7fb760a35000-7fb760c35000 ---p 00029000 08:23 29759929                   /usr/lib64/libpng15.so.15.13.0
7fb760c35000-7fb760c36000 r--p 00029000 08:23 29759929                   /usr/lib64/libpng15.so.15.13.0
7fb760c36000-7fb760c37000 rw-p 0002a000 08:23 29759929                   /usr/lib64/libpng15.so.15.13.0
7fb760c37000-7fb7631b5000 r-xp 00000000 08:23 2888877                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcurand.so.10.1.2.89
7fb7631b5000-7fb7633b4000 ---p 0257e000 08:23 2888877                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcurand.so.10.1.2.89
7fb7633b4000-7fb764783000 rw-p 0257d000 08:23 2888877                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcurand.so.10.1.2.89
7fb764783000-7fb764cda000 rw-p 00000000 00:00 0 
7fb764cda000-7fb764ee7000 r-xp 00000000 08:23 25699833                   /opt/relion-3.1.3/lib/libfftw3f.so.3.5.8
7fb764ee7000-7fb7650e6000 ---p 0020d000 08:23 25699833                   /opt/relion-3.1.3/lib/libfftw3f.so.3.5.8
7fb7650e6000-7fb7650f7000 r--p 0020c000 08:23 25699833                   /opt/relion-3.1.3/lib/libfftw3f.so.3.5.8
7fb7650f7000-7fb7650f8000 rw-p 0021d000 08:23 25699833                   /opt/relion-3.1.3/lib/libfftw3f.so.3.5.8
7fb7650f8000-7fb76527d000 r-xp 00000000 08:23 25699835                   /opt/relion-3.1.3/lib/libfftw3.so.3.5.8
7fb76527d000-7fb76547c000 ---p 00185000 08:23 25699835                   /opt/relion-3.1.3/lib/libfftw3.so.3.5.8
7fb76547c000-7fb765488000 r--p 00184000 08:23 25699835                   /opt/relion-3.1.3/lib/libfftw3.so.3.5.8
7fb765488000-7fb765489000 rw-p 00190000 08:23 25699835                   /opt/relion-3.1.3/lib/libfftw3.so.3.5.8
7fb765489000-7fb7654f8000 r-xp 00000000 08:23 29768750                   /usr/lib64/libtiff.so.5.2.0
7fb7654f8000-7fb7656f8000 ---p 0006f000 08:23 29768750                   /usr/lib64/libtiff.so.5.2.0
7fb7656f8000-7fb7656f9000 r--p 0006f000 08:23 29768750                   /usr/lib64/libtiff.so.5.2.0
7fb7656f9000-7fb7656fc000 rw-p 00070000 08:23 29768750                   /usr/lib64/libtiff.so.5.2.0
7fb7656fc000-7fb7656fd000 rw-p 00000000 00:00 0 
7fb7656fd000-7fb7656ff000 r-xp 00000000 08:23 29759171                   /usr/lib64/libdl-2.17.so
7fb7656ff000-7fb7658ff000 ---p 00002000 08:23 29759171                   /usr/lib64/libdl-2.17.so
7fb7658ff000-7fb765900000 r--p 00002000 08:23 29759171                   /usr/lib64/libdl-2.17.so
7fb765900000-7fb765901000 rw-p 00003000 08:23 29759171                   /usr/lib64/libdl-2.17.so
7fb765901000-7fb7665d4000 r-xp 00000000 08:23 5523962                    /opt/mpich-3.4.1/lib/libmpi.so.12.1.10
7fb7665d4000-7fb7667d4000 ---p 00cd3000 08:23 5523962                    /opt/mpich-3.4.1/lib/libmpi.so.12.1.10
7fb7667d4000-7fb7667fb000 r--p 00cd3000 08:23 5523962                    /opt/mpich-3.4.1/lib/libmpi.so.12.1.10
7fb7667fb000-7fb766808000 rw-p 00cfa000 08:23 5523962                    /opt/mpich-3.4.1/lib/libmpi.so.12.1.10
7fb766808000-7fb766867000 rw-p 00000000 00:00 0 
7fb766867000-7fb766885000 r-xp 00000000 08:23 5523970                    /opt/mpich-3.4.1/lib/libmpicxx.so.12.1.10
7fb766885000-7fb766a85000 ---p 0001e000 08:23 5523970                    /opt/mpich-3.4.1/lib/libmpicxx.so.12.1.10
7fb766a85000-7fb766a87000 r--p 0001e000 08:23 5523970                    /opt/mpich-3.4.1/lib/libmpicxx.so.12.1.10
7fb766a87000-7fb766a88000 rw-p 00020000 08:23 5523970                    /opt/mpich-3.4.1/lib/libmpicxx.so.12.1.10
7fb766a88000-7fb76fc9e000 r-xp 00000000 08:23 2888874                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcufft.so.10.1.2.89
7fb76fc9e000-7fb76fe9d000 ---p 09216000 08:23 2888874                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcufft.so.10.1.2.89
7fb76fe9d000-7fb76feaf000 rw-p 09215000 08:23 2888874                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcufft.so.10.1.2.89
7fb76feaf000-7fb76ff28000 rw-p 00000000 00:00 0 
7fb76ff28000-7fb76ff4a000 r-xp 00000000 08:23 29758944                   /usr/lib64/ld-2.17.so
7fb770101000-7fb770116000 rw-p 00000000 00:00 0 
7fb770146000-7fb770149000 rw-p 00000000 00:00 0 
7fb770149000-7fb77014a000 r--p 00021000 08:23 29758944                   /usr/lib64/ld-2.17.so
7fb77014a000-7fb77014b000 rw-p 00022000 08:23 29758944                   /usr/lib64/ld-2.17.so
7fb77014b000-7fb77014c000 rw-p 00000000 00:00 0 
7ffdc8d52000-7ffdc8d74000 rw-p 00000000 00:00 0                          [stack]
7ffdc8dc6000-7ffdc8dc8000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
*** Error in `/opt/relion-3.1.3/bin/relion_refine_mpi': corrupted size vs. prev_size: 0x00000000013536d0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x80d97)[0x7f825a139d97]
/lib64/libc.so.6(+0x8136e)[0x7f825a13a36e]
/lib64/libnvidia-fatbinaryloader.so.430.26(+0x1e18a)[0x7f825701718a]
/lib64/libnvidia-fatbinaryloader.so.430.26(+0x1ba2c)[0x7f8257014a2c]
/lib64/libnvidia-fatbinaryloader.so.430.26(+0x1170a)[0x7f825700a70a]
/lib64/libnvidia-fatbinaryloader.so.430.26(+0x1b2a1)[0x7f82570142a1]
/lib64/libnvidia-fatbinaryloader.so.430.26(+0x27c3d)[0x7f8257020c3d]
/lib64/libc.so.6(__cxa_finalize+0x9a)[0x7f825a0f2eda]
/lib64/libnvidia-fatbinaryloader.so.430.26(+0x5cb6)[0x7f8256ffecb6]
======= Memory map: ========
00400000-00a24000 r-xp 00000000 08:23 25699894                           /opt/relion-3.1.3/bin/relion_refine_mpi
00c23000-00c24000 r--p 00623000 08:23 25699894                           /opt/relion-3.1.3/bin/relion_refine_mpi
00c24000-00c25000 rw-p 00624000 08:23 25699894                           /opt/relion-3.1.3/bin/relion_refine_mpi
00c25000-00c65000 rw-p 00000000 00:00 0 
0119a000-aba0a000 rw-p 00000000 00:00 0                                  [heap]
200000000-c00200000 ---p 00000000 00:00 0 
7f81e8000000-7f81e8021000 rw-p 00000000 00:00 0 
7f81e8021000-7f81ec000000 ---p 00000000 00:00 0 
7f81f0000000-7f81f0021000 rw-p 00000000 00:00 0 
7f81f0021000-7f81f4000000 ---p 00000000 00:00 0 
7f8230000000-7f8240000000 ---p 00000000 00:00 0 
7f82400b9000-7f82400ba000 ---p 00000000 00:00 0 
7f82400ba000-7f82408ba000 rw-p 00000000 00:00 0 
7f82408ba000-7f82408bb000 ---p 00000000 00:00 0 
7f82408bb000-7f82410bb000 rw-p 00000000 00:00 0 
7f824a075000-7f824a07d000 rw-s 00000000 00:13 306593                     /dev/shm/mpich_shar_tmpyBqGKe (deleted)
7f824a07d000-7f824a07e000 rw-s 00000000 00:13 306592                     /dev/shm/mpich_shar_tmp9czlFN (deleted)
7f824a084000-7f824a085000 ---p 00000000 00:00 0 
7f824a085000-7f824a8af000 rw-p 00000000 00:00 0 
7f824a8af000-7f82508af000 ---p 00000000 00:00 0 
7f82508af000-7f82508bf000 -w-s 00000000 00:05 184432                     /dev/nvidia9
7f82508bf000-7f82508cf000 -w-s 00000000 00:05 180269                     /dev/nvidia8
7f82508cf000-7f82508df000 -w-s 00000000 00:05 102016                     /dev/nvidia7
7f82508df000-7f82508ef000 -w-s 00000000 00:05 175290                     /dev/nvidia6
7f82508ef000-7f82508ff000 -w-s 00000000 00:05 170607                     /dev/nvidia5
7f82508ff000-7f825090f000 -w-s 00000000 00:05 170507                     /dev/nvidia4
7f825090f000-7f825091f000 -w-s 00000000 00:05 167051                     /dev/nvidia3
7f825091f000-7f825092f000 -w-s 00000000 00:05 170403                     /dev/nvidia2
7f825092f000-7f825093f000 -w-s 00000000 00:05 159837                     /dev/nvidia1
7f825093f000-7f825094f000 -w-s 00000000 00:05 148719                     /dev/nvidia0
7f8251097000-7f8251098000 ---p 00000000 00:00 0 
7f8251098000-7f8251898000 rw-p 00000000 00:00 0 
7f8251898000-7f8251899000 ---p 00000000 00:00 0 
7f8251899000-7f8252099000 rw-p 00000000 00:00 0 
7f8252099000-7f82520ec000 r-xp 00000000 08:23 11289130                   /home/spuser/miniconda3/lib/libfabric/prov/librxm-fi.so
7f82520ec000-7f82522eb000 ---p 00053000 08:23 11289130                   /home/spuser/miniconda3/lib/libfabric/prov/librxm-fi.so
7f82522eb000-7f82522ef000 rw-p 00052000 08:23 11289130                   /home/spuser/miniconda3/lib/libfabric/prov/librxm-fi.so
7f82522ef000-7f825234e000 r-xp 00000000 08:23 29780055                   /usr/lib64/libpsm2.so.2.1
7f825234e000-7f825254e000 ---p 0005f000 08:23 29780055                   /usr/lib64/libpsm2.so.2.1
7f825254e000-7f825254f000 r--p 0005f000 08:23 29780055                   /usr/lib64/libpsm2.so.2.1
7f825254f000-7f8252550000 rw-p 00060000 08:23 29780055                   /usr/lib64/libpsm2.so.2.1
7f8252550000-7f8252552000 rw-p 00000000 00:00 0 
7f8252552000-7f82525ca000 r-xp 00000000 08:23 11289134                   /home/spuser/miniconda3/lib/libfabric/prov/libpsmx2-fi.so
7f82525ca000-7f82527c9000 ---p 00078000 08:23 11289134                   /home/spuser/miniconda3/lib/libfabric/prov/libpsmx2-fi.so
7f82527c9000-7f82527cf000 rw-p 00077000 08:23 11289134                   /home/spuser/miniconda3/lib/libfabric/prov/libpsmx2-fi.so
7f82527cf000-7f82527d3000 rw-p 00000000 00:00 0 
7f82527d3000-7f825282d000 r-xp 00000000 08:23 11289132                   /home/spuser/miniconda3/lib/libfabric/prov/libsockets-fi.so
7f825282d000-7f8252a2d000 ---p 0005a000 08:23 11289132                   /home/spuser/miniconda3/lib/libfabric/prov/libsockets-fi.so
7f8252a2d000-7f8252a31000 rw-p 0005a000 08:23 11289132                   /home/spuser/miniconda3/lib/libfabric/prov/libsockets-fi.so
7f8252a31000-7f8252a74000 r-xp 00000000 08:23 11289128                   /home/spuser/miniconda3/lib/libfabric/prov/libtcp-fi.so
7f8252a74000-7f8252c74000 ---p 00043000 08:23 11289128                   /home/spuser/miniconda3/lib/libfabric/prov/libtcp-fi.so
7f8252c74000-7f8252c77000 rw-p 00043000 08:23 11289128                   /home/spuser/miniconda3/lib/libfabric/prov/libtcp-fi.so
7f8252c77000-7f8252c81000 r-xp 00000000 08:23 29760907                   /usr/lib64/libnuma.so.1
7f8252c81000-7f8252e81000 ---p 0000a000 08:23 29760907                   /usr/lib64/libnuma.so.1
7f8252e81000-7f8252e82000 r--p 0000a000 08:23 29760907                   /usr/lib64/libnuma.so.1
7f8252e82000-7f8252e83000 rw-p 0000b000 08:23 29760907                   /usr/lib64/libnuma.so.1
7f8252e83000-7f8252fb3000 r-xp 00000000 08:23 11289135                   /home/spuser/miniconda3/lib/libfabric/prov/libpsm3-fi.so
7f8252fb3000-7f82531b2000 ---p 00130000 08:23 11289135                   /home/spuser/miniconda3/lib/libfabric/prov/libpsm3-fi.so
7f82531b2000-7f82531b9000 rw-p 0012f000 08:23 11289135                   /home/spuser/miniconda3/lib/libfabric/prov/libpsm3-fi.so
7f82531b9000-7f82531df000 rw-p 00000000 00:00 0 
7f82531df000-7f8253225000 r-xp 00000000 08:23 11289113                   /home/spuser/miniconda3/lib/libfabric/libfabric.so.1
7f8253225000-7f8253425000 ---p 00046000 08:23 11289113                   /home/spuser/miniconda3/lib/libfabric/libfabric.so.1
7f8253425000-7f8253428000 rw-p 00046000 08:23 11289113                   /home/spuser/miniconda3/lib/libfabric/libfabric.so.1
7f8253428000-7f8253474000 r-xp 00000000 08:23 11289129                   /home/spuser/miniconda3/lib/libfabric/prov/libshm-fi.so
7f8253474000-7f8253673000 ---p 0004c000 08:23 11289129                   /home/spuser/miniconda3/lib/libfabric/prov/libshm-fi.so
7f8253673000-7f8253677000 rw-p 0004b000 08:23 11289129                   /home/spuser/miniconda3/lib/libfabric/prov/libshm-fi.so
7f825653e000-7f825654d000 r-xp 00000000 08:23 29759940                   /usr/lib64/libbz2.so.1.0.6
7f825654d000-7f825674c000 ---p 0000f000 08:23 29759940                   /usr/lib64/libbz2.so.1.0.6
7f825674c000-7f825674d000 r--p 0000e000 08:23 29759940                   /usr/lib64/libbz2.so.1.0.6
7f825674d000-7f825674e000 rw-p 0000f000 08:23 29759940                   /usr/lib64/libbz2.so.1.0.6
7f825674e000-7f8256765000 r-xp 00000000 08:23 29776643                   /usr/lib64/libelf-0.172.so
7f8256765000-7f8256964000 ---p 00017000 08:23 29776643                   /usr/lib64/libelf-0.172.so
7f8256964000-7f8256965000 r--p 00016000 08:23 29776643                   /usr/lib64/libelf-0.172.so
7f8256965000-7f8256966000 rw-p 00017000 08:23 29776643                   /usr/lib64/libelf-0.172.so
7f8256966000-7f825696a000 r-xp 00000000 08:23 29759596                   /usr/lib64/libattr.so.1.1.0
7f825696a000-7f8256b69000 ---p 00004000 08:23 29759596                   /usr/lib64/libattr.so.1.1.0
7f8256b69000-7f8256b6a000 r--p 00003000 08:23 29759596                   /usr/lib64/libattr.so.1.1.0
7f8256b6a000-7f8256b6b000 rw-p 00004000 08:23 29759596                   /usr/lib64/libattr.so.1.1.0
7f8256b6b000-7f8256b89000 r-xp 00000000 08:23 29759988                   /usr/lib64/libnl-3.so.200.23.0
7f8256b89000-7f8256d89000 ---p 0001e000 08:23 29759988                   /usr/lib64/libnl-3.so.200.23.0
7f8256d89000-7f8256d8b000 r--p 0001e000 08:23 29759988                   /usr/lib64/libnl-3.so.200.23.0
7f8256d8b000-7f8256d8c000 rw-p 00020000 08:23 29759988                   /usr/lib64/libnl-3.so.200.23.0
7f8256d8c000-7f8256df0000 r-xp 00000000 08:23 29759996                   /usr/lib64/libnl-route-3.so.200.23.0
7f8256df0000-7f8256fef000 ---p 00064000 08:23 29759996                   /usr/lib64/libnl-route-3.so.200.23.0
7f8256fef000-7f8256ff2000 r--p 00063000 08:23 29759996                   /usr/lib64/libnl-route-3.so.200.23.0
7f8256ff2000-7f8256ff7000 rw-p 00066000 08:23 29759996                   /usr/lib64/libnl-route-3.so.200.23.0
7f8256ff7000-7f8256ff9000 rw-p 00000000 00:00 0 
7f8256ff9000-7f8257040000 r-xp 00000000 08:23 29761878                   /usr/lib64/libnvidia-fatbinaryloader.so.430.26
7f8257040000-7f8257240000 ---p 00047000 08:23 29761878                   /usr/lib64/libnvidia-fatbinaryloader.so.430.26
7f8257240000-7f8257242000 rw-p 00047000 08:23 29761878                   /usr/lib64/libnvidia-fatbinaryloader.so.430.26
7f8257242000-7f8257247000 rw-p 00000000 00:00 0 
7f8257247000-7f825726c000 r-xp 00000000 08:23 29759932                   /usr/lib64/liblzma.so.5.2.2
7f825726c000-7f825746b000 ---p 00025000 08:23 29759932                   /usr/lib64/liblzma.so.5.2.2
7f825746b000-7f825746c000 r--p 00024000 08:23 29759932                   /usr/lib64/liblzma.so.5.2.2
7f825746c000-7f825746d000 rw-p 00025000 08:23 29759932                   /usr/lib64/liblzma.so.5.2.2
7f825746d000-7f82574b9000 r-xp 00000000 08:23 29765135                   /usr/lib64/libdw-0.172.so
7f82574b9000-7f82576b9000 ---p 0004c000 08:23 29765135                   /usr/lib64/libdw-0.172.so
7f82576b9000-7f82576bb000 r--p 0004c000 08:23 29765135                   /usr/lib64/libdw-0.172.so
7f82576bb000-7f82576bc000 rw-p 0004e000 08:23 29765135                   /usr/lib64/libdw-0.172.so
7f82576bc000-7f82576c0000 r-xp 00000000 08:23 29759598                   /usr/lib64/libcap.so.2.22
7f82576c0000-7f82578bf000 ---p 00004000 08:23 29759598                   /usr/lib64/libcap.so.2.22
7f82578bf000-7f82578c0000 r--p 00003000 08:23 29759598                   /usr/lib64/libcap.so.2.22
7f82578c0000-7f82578c1000 rw-p 00004000 08:23 29759598                   /usr/lib64/libcap.so.2.22
7f82578c1000-7f82578d6000 r-xp 00000000 08:23 29759457                   /usr/lib64/libz.so.1.2.7
7f82578d6000-7f8257ad5000 ---p 00015000 08:23 29759457                   /usr/lib64/libz.so.1.2.7
7f8257ad5000-7f8257ad6000 r--p 00014000 08:23 29759457                   /usr/lib64/libz.so.1.2.7
7f8257ad6000-7f8257ad7000 rw-p 00015000 08:23 29759457                   /usr/lib64/libz.so.1.2.7
7f8257ad7000-7f8257b1a000 r-xp 00000000 08:23 29759926                   /usr/lib64/libjpeg.so.62.1.0
7f8257b1a000-7f8257d1a000 ---p 00043000 08:23 29759926                   /usr/lib64/libjpeg.so.62.1.0
7f8257d1a000-7f8257d1b000 r--p 00043000 08:23 29759926                   /usr/lib64/libjpeg.so.62.1.0
7f8257d1b000-7f8257d1c000 rw-p 00044000 08:23 29759926                   /usr/lib64/libjpeg.so.62.1.0
7f8257d1c000-7f8257d2c000 rw-p 00000000 00:00 0 
7f8257d2c000-7f8257d35000 r-xp 00000000 08:23 29768747                   /usr/lib64/libjbig.so.2.0
7f8257d35000-7f8257f34000 ---p 00009000 08:23 29768747                   /usr/lib64/libjbig.so.2.0
7f8257f34000-7f8257f35000 r--p 00008000 08:23 29768747                   /usr/lib64/libjbig.so.2.0
7f8257f35000-7f8257f38000 rw-p 00009000 08:23 29768747                   /usr/lib64/libjbig.so.2.0
7f8257f38000-7f8257f3e000 r-xp 00000000 08:23 3162208                    /opt/gcc_7.5.0/lib64/libatomic.so.1.2.0
7f8257f3e000-7f825813d000 ---p 00006000 08:23 3162208                    /opt/gcc_7.5.0/lib64/libatomic.so.1.2.0
7f825813d000-7f825813e000 r--p 00005000 08:23 3162208                    /opt/gcc_7.5.0/lib64/libatomic.so.1.2.0
7f825813e000-7f825813f000 rw-p 00006000 08:23 3162208                    /opt/gcc_7.5.0/lib64/libatomic.so.1.2.0
7f825813f000-7f8258140000 rw-p 00000000 00:00 0 
7f8258140000-7f825818d000 r-xp 00000000 08:23 29762881                   /usr/lib64/libnl.so.1.1.4
7f825818d000-7f825838d000 ---p 0004d000 08:23 29762881                   /usr/lib64/libnl.so.1.1.4
7f825838d000-7f825838e000 r--p 0004d000 08:23 29762881                   /usr/lib64/libnl.so.1.1.4
7f825838e000-7f8258393000 rw-p 0004e000 08:23 29762881                   /usr/lib64/libnl.so.1.1.4
7f8258393000-7f82583a8000 r-xp 00000000 08:23 29766308                   /usr/lib64/libibverbs.so.1.1.17.2
7f82583a8000-7f82585a8000 ---p 00015000 08:23 29766308                   /usr/lib64/libibverbs.so.1.1.17.2
7f82585a8000-7f82585a9000 r--p 00015000 08:23 29766308                   /usr/lib64/libibverbs.so.1.1.17.2
7f82585a9000-7f82585aa000 rw-p 00016000 08:23 29766308                   /usr/lib64/libibverbs.so.1.1.17.2
7f82585aa000-7f82585bf000 r-xp 00000000 08:23 29766998                   /usr/lib64/librdmacm.so.1.1.17.2
7f82585bf000-7f82587be000 ---p 00015000 08:23 29766998                   /usr/lib64/librdmacm.so.1.1.17.2
7f82587be000-7f82587bf000 r--p 00014000 08:23 29766998                   /usr/lib64/librdmacm.so.1.1.17.2
7f82587bf000-7f82587c0000 rw-p 00015000 08:23 29766998                   /usr/lib64/librdmacm.so.1.1.17.2
7f82587c0000-7f82587c1000 rw-p 00000000 00:00 0 
7f82587c1000-7f82595a9000 r-xp 00000000 08:23 29760290                   /usr/lib64/libcuda.so.430.26
7f82595a9000-7f82597a9000 ---p 00de8000 08:23 29760290                   /usr/lib64/libcuda.so.430.26
7f82597a9000-7f8259921000 rw-p 00de8000 08:23 29760290                   /usr/lib64/libcuda.so.430.26
7f8259921000-7f8259931000 rw-p 00000000 00:00 0 
7f8259931000-7f8259a90000 r-xp 00000000 08:23 29759936                   /usr/lib64/libxml2.so.2.9.1
7f8259a90000-7f8259c8f000 ---p 0015f000 08:23 29759936                   /usr/lib64/libxml2.so.2.9.1
7f8259c8f000-7f8259c97000 r--p 0015e000 08:23 29759936                   /usr/lib64/libxml2.so.2.9.1
7f8259c97000-7f8259c99000 rw-p 00166000 08:23 29759936                   /usr/lib64/libxml2.so.2.9.1
7f8259c99000-7f8259c9b000 rw-p 00000000 00:00 0 
7f8259c9b000-7f8259cb0000 r-xp 00000000 08:23 29765150                   /usr/lib64/libudev.so.1.6.2
7f8259cb0000-7f8259eaf000 ---p 00015000 08:23 29765150                   /usr/lib64/libudev.so.1.6.2
7f8259eaf000-7f8259eb0000 r--p 00014000 08:23 29765150                   /usr/lib64/libudev.so.1.6.2
7f8259eb0000-7f8259eb1000 rw-p 00015000 08:23 29765150                   /usr/lib64/libudev.so.1.6.2
7f8259eb1000-7f8259eb8000 r-xp 00000000 08:23 29759195                   /usr/lib64/librt-2.17.so
7f8259eb8000-7f825a0b7000 ---p 00007000 08:23 29759195                   /usr/lib64/librt-2.17.so
7f825a0b7000-7f825a0b8000 r--p 00006000 08:23 29759195                   /usr/lib64/librt-2.17.so
7f825a0b8000-7f825a0b9000 rw-p 00007000 08:23 29759195                   /usr/lib64/librt-2.17.so
7f825a0b9000-7f825a27b000 r-xp 00000000 08:23 29759165                   /usr/lib64/libc-2.17.so
7f825a27b000-7f825a47b000 ---p 001c2000 08:23 29759165                   /usr/lib64/libc-2.17.so
7f825a47b000-7f825a47f000 r--p 001c2000 08:23 29759165                   /usr/lib64/libc-2.17.so
7f825a47f000-7f825a481000 rw-p 001c6000 08:23 29759165                   /usr/lib64/libc-2.17.so
7f825a481000-7f825a486000 rw-p 00000000 00:00 0 
7f825a486000-7f825a49d000 r-xp 00000000 08:23 29759191                   /usr/lib64/libpthread-2.17.so
7f825a49d000-7f825a69c000 ---p 00017000 08:23 29759191                   /usr/lib64/libpthread-2.17.so
7f825a69c000-7f825a69d000 r--p 00016000 08:23 29759191                   /usr/lib64/libpthread-2.17.so
7f825a69d000-7f825a69e000 rw-p 00017000 08:23 29759191                   /usr/lib64/libpthread-2.17.so
7f825a69e000-7f825a6a2000 rw-p 00000000 00:00 0 
7f825a6a2000-7f825a6b8000 r-xp 00000000 08:23 3161869                    /opt/gcc_7.5.0/lib64/libgcc_s.so.1
7f825a6b8000-7f825a8b7000 ---p 00016000 08:23 3161869                    /opt/gcc_7.5.0/lib64/libgcc_s.so.1
7f825a8b7000-7f825a8b8000 r--p 00015000 08:23 3161869                    /opt/gcc_7.5.0/lib64/libgcc_s.so.1
7f825a8b8000-7f825a8b9000 rw-p 00016000 08:23 3161869                    /opt/gcc_7.5.0/lib64/libgcc_s.so.1
7f825a8b9000-7f825a8e6000 r-xp 00000000 08:23 3162222                    /opt/gcc_7.5.0/lib64/libgomp.so.1.0.0
7f825a8e6000-7f825aae5000 ---p 0002d000 08:23 3162222                    /opt/gcc_7.5.0/lib64/libgomp.so.1.0.0
7f825aae5000-7f825aae6000 r--p 0002c000 08:23 3162222                    /opt/gcc_7.5.0/lib64/libgomp.so.1.0.0
7f825aae6000-7f825aae7000 rw-p 0002d000 08:23 3162222                    /opt/gcc_7.5.0/lib64/libgomp.so.1.0.0
7f825aae7000-7f825abe8000 r-xp 00000000 08:23 29759173                   /usr/lib64/libm-2.17.so
7f825abe8000-7f825ade7000 ---p 00101000 08:23 29759173                   /usr/lib64/libm-2.17.so
7f825ade7000-7f825ade8000 r--p 00100000 08:23 29759173                   /usr/lib64/libm-2.17.so
7f825ade8000-7f825ade9000 rw-p 00101000 08:23 29759173                   /usr/lib64/libm-2.17.so
7f825ade9000-7f825af5c000 r-xp 00000000 08:23 3162144                    /opt/gcc_7.5.0/lib64/libstdc++.so.6.0.24
7f825af5c000-7f825b15c000 ---p 00173000 08:23 3162144                    /opt/gcc_7.5.0/lib64/libstdc++.so.6.0.24
7f825b15c000-7f825b166000 r--p 00173000 08:23 3162144                    /opt/gcc_7.5.0/lib64/libstdc++.so.6.0.24
7f825b166000-7f825b168000 rw-p 0017d000 08:23 3162144                    /opt/gcc_7.5.0/lib64/libstdc++.so.6.0.24
7f825b168000-7f825b16c000 rw-p 00000000 00:00 0 
7f825b16c000-7f825b1e5000 r-xp 00000000 08:23 2888908                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2.89
7f825b1e5000-7f825b3e4000 ---p 00079000 08:23 2888908                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2.89
7f825b3e4000-7f825b3e8000 rw-p 00078000 08:23 2888908                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudart.so.10.2.89
7f825b3e8000-7f825b3ea000 rw-p 00000000 00:00 0 
7f825b3ea000-7f825b413000 r-xp 00000000 08:23 29759929                   /usr/lib64/libpng15.so.15.13.0
7f825b413000-7f825b613000 ---p 00029000 08:23 29759929                   /usr/lib64/libpng15.so.15.13.0
7f825b613000-7f825b614000 r--p 00029000 08:23 29759929                   /usr/lib64/libpng15.so.15.13.0
7f825b614000-7f825b615000 rw-p 0002a000 08:23 29759929                   /usr/lib64/libpng15.so.15.13.0
7f825b615000-7f825db93000 r-xp 00000000 08:23 2888877                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcurand.so.10.1.2.89
7f825db93000-7f825dd92000 ---p 0257e000 08:23 2888877                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcurand.so.10.1.2.89
7f825dd92000-7f825f161000 rw-p 0257d000 08:23 2888877                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcurand.so.10.1.2.89
7f825f161000-7f825f6b8000 rw-p 00000000 00:00 0 
7f825f6b8000-7f825f8c5000 r-xp 00000000 08:23 25699833                   /opt/relion-3.1.3/lib/libfftw3f.so.3.5.8
7f825f8c5000-7f825fac4000 ---p 0020d000 08:23 25699833                   /opt/relion-3.1.3/lib/libfftw3f.so.3.5.8
7f825fac4000-7f825fad5000 r--p 0020c000 08:23 25699833                   /opt/relion-3.1.3/lib/libfftw3f.so.3.5.8
7f825fad5000-7f825fad6000 rw-p 0021d000 08:23 25699833                   /opt/relion-3.1.3/lib/libfftw3f.so.3.5.8
7f825fad6000-7f825fc5b000 r-xp 00000000 08:23 25699835                   /opt/relion-3.1.3/lib/libfftw3.so.3.5.8
7f825fc5b000-7f825fe5a000 ---p 00185000 08:23 25699835                   /opt/relion-3.1.3/lib/libfftw3.so.3.5.8
7f825fe5a000-7f825fe66000 r--p 00184000 08:23 25699835                   /opt/relion-3.1.3/lib/libfftw3.so.3.5.8
7f825fe66000-7f825fe67000 rw-p 00190000 08:23 25699835                   /opt/relion-3.1.3/lib/libfftw3.so.3.5.8
7f825fe67000-7f825fed6000 r-xp 00000000 08:23 29768750                   /usr/lib64/libtiff.so.5.2.0
7f825fed6000-7f82600d6000 ---p 0006f000 08:23 29768750                   /usr/lib64/libtiff.so.5.2.0
7f82600d6000-7f82600d7000 r--p 0006f000 08:23 29768750                   /usr/lib64/libtiff.so.5.2.0
7f82600d7000-7f82600da000 rw-p 00070000 08:23 29768750                   /usr/lib64/libtiff.so.5.2.0
7f82600da000-7f82600db000 rw-p 00000000 00:00 0 
7f82600db000-7f82600dd000 r-xp 00000000 08:23 29759171                   /usr/lib64/libdl-2.17.so
7f82600dd000-7f82602dd000 ---p 00002000 08:23 29759171                   /usr/lib64/libdl-2.17.so
7f82602dd000-7f82602de000 r--p 00002000 08:23 29759171                   /usr/lib64/libdl-2.17.so
7f82602de000-7f82602df000 rw-p 00003000 08:23 29759171                   /usr/lib64/libdl-2.17.so
7f82602df000-7f8260fb2000 r-xp 00000000 08:23 5523962                    /opt/mpich-3.4.1/lib/libmpi.so.12.1.10
7f8260fb2000-7f82611b2000 ---p 00cd3000 08:23 5523962                    /opt/mpich-3.4.1/lib/libmpi.so.12.1.10
7f82611b2000-7f82611d9000 r--p 00cd3000 08:23 5523962                    /opt/mpich-3.4.1/lib/libmpi.so.12.1.10
7f82611d9000-7f82611e6000 rw-p 00cfa000 08:23 5523962                    /opt/mpich-3.4.1/lib/libmpi.so.12.1.10
7f82611e6000-7f8261245000 rw-p 00000000 00:00 0 
7f8261245000-7f8261263000 r-xp 00000000 08:23 5523970                    /opt/mpich-3.4.1/lib/libmpicxx.so.12.1.10
7f8261263000-7f8261463000 ---p 0001e000 08:23 5523970                    /opt/mpich-3.4.1/lib/libmpicxx.so.12.1.10
7f8261463000-7f8261465000 r--p 0001e000 08:23 5523970                    /opt/mpich-3.4.1/lib/libmpicxx.so.12.1.10
7f8261465000-7f8261466000 rw-p 00020000 08:23 5523970                    /opt/mpich-3.4.1/lib/libmpicxx.so.12.1.10
7f8261466000-7f826a67c000 r-xp 00000000 08:23 2888874                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcufft.so.10.1.2.89
7f826a67c000-7f826a87b000 ---p 09216000 08:23 2888874                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcufft.so.10.1.2.89
7f826a87b000-7f826a88d000 rw-p 09215000 08:23 2888874                    /usr/local/cuda-10.2/targets/x86_64-linux/lib/libcufft.so.10.1.2.89
7f826a88d000-7f826a906000 rw-p 00000000 00:00 0 
7f826a906000-7f826a928000 r-xp 00000000 08:23 29758944                   /usr/lib64/ld-2.17.so
7f826aadf000-7f826aaf4000 rw-p 00000000 00:00 0 
7f826ab24000-7f826ab27000 rw-p 00000000 00:00 0 
7f826ab27000-7f826ab28000 r--p 00021000 08:23 29758944                   /usr/lib64/ld-2.17.so
7f826ab28000-7f826ab29000 rw-p 00022000 08:23 29758944                   /usr/lib64/ld-2.17.so
7f826ab29000-7f826ab2a000 rw-p 00000000 00:00 0 
7ffc2a548000-7ffc2a56a000 rw-p 00000000 00:00 0                          [stack]
7ffc2a5fc000-7ffc2a5fe000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

[biochem-fan]

The error looks like a CUDA runtime problem. Are you sure your compilation process was fine? Are you using the right version of runtime compatible with your binary?

What happens if you re-compile 3.1.1?

[heejongkim]

I have been using the same binary since 3.1.2 release and it only started appearing since a week or two ago suddenly with same dependencies.

Should I recompile them (3.1.1 and 3.1.2 and 3.1.3) and test them out?

Any recommended CUDA version?

[biochem-fan]

Should I recompile them (3.1.1 and 3.1.2 and 3.1.3) and test them out?

Yes, please.

Any recommended CUDA version?

Nothing in particular.

[heejongkim]

Recompiling 3.1.3 with lower version of gcc from 7.5.0 to 4.8.5 and corresponding openmpi version fixed the issue tentatively. (I suspect that manually compiled MPICH 3.4.1 with gcc 7.5.0 might've caused the issue)

Is there a page listing the dependencies/compilers and their versions that I've been missing?

Thank you.

[biochem-fan]

I don't know why MPICH with GCC 7.5.0 does not work. We mostly work on GCC or Intel Compiler with OpenMPI. Other combinations should work but we ourselves cannot test or investigate them due to limited resources.

[heejongkim]

I will try a few combinations with different gcc with openMPI and MPICH.

[alburse]

I encountered this error recently and solved it by removing duplicate particles from the particles.star file after failing to solve it by installing all combinations of GCC and OpenMPI.