dkimanius
commented
6 years ago Looks like something happened with the scratch-drive file-access, considering the error:
master encountered error: ERROR: Cannot read file /tmp/isomg01/relion_volatile/particles.mrcs...
Make sure that the scratch-drive is properly connected and mounted. You should be able to continue from iteration 10 using the continue-flag.
ncoudray
Hi,
One of our 2D classification crashed after 10 iterartions. I haven't found similar post yet, sorry if I missed it. Here is the run.err:
master encountered error: ERROR: Cannot read file /tmp/isomg01/relion_volatile/particles.mrcs It does not exist File: /local/apps/relion/2.1.0/src/image.h line: 157 rm: cannot remove ‘/tmp/4120472.1.gpu1.q/machines’: No such file or directory rm: cannot remove ‘/tmp/4120472.1.gpu1.q/ssh’: No such file or directoryand the run.out: `-catch_rsh /cm/local/apps/sge/var/spool/gpu102/active_jobs/4120472.1/pe_hostfile gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 gpu102 Warning: no access to tty (Bad file descriptor). Thus no job control in this shell. === RELION MPI setup ===
Slave 7 runs on host = gpu102
uniqueHost gpu102 has 7 ranks. GPU-ids not specified for this rank, threads will automatically be mapped to available devices. Thread 0 on slave 1 mapped to device 0 Thread 1 on slave 1 mapped to device 0 Thread 2 on slave 1 mapped to device 0 Thread 3 on slave 1 mapped to device 0 Thread 4 on slave 1 mapped to device 0 Thread 5 on slave 1 mapped to device 0 GPU-ids not specified for this rank, threads will automatically be mapped to available devices. Thread 0 on slave 2 mapped to device 1 Thread 1 on slave 2 mapped to device 1 Thread 2 on slave 2 mapped to device 1 Thread 3 on slave 2 mapped to device 1 Thread 4 on slave 2 mapped to device 1 Thread 5 on slave 2 mapped to device 2 GPU-ids not specified for this rank, threads will automatically be mapped to available devices. Thread 0 on slave 3 mapped to device 2 Thread 1 on slave 3 mapped to device 2 Thread 2 on slave 3 mapped to device 2 Thread 3 on slave 3 mapped to device 2 Thread 4 on slave 3 mapped to device 3 Thread 5 on slave 3 mapped to device 3 GPU-ids not specified for this rank, threads will automatically be mapped to available devices. Thread 0 on slave 4 mapped to device 3 Thread 1 on slave 4 mapped to device 3 Thread 2 on slave 4 mapped to device 3 Thread 3 on slave 4 mapped to device 4 Thread 4 on slave 4 mapped to device 4 Thread 5 on slave 4 mapped to device 4 GPU-ids not specified for this rank, threads will automatically be mapped to available devices. Thread 0 on slave 5 mapped to device 4 Thread 1 on slave 5 mapped to device 4 Thread 2 on slave 5 mapped to device 4 Thread 3 on slave 5 mapped to device 5 Thread 4 on slave 5 mapped to device 5 Thread 5 on slave 5 mapped to device 5 GPU-ids not specified for this rank, threads will automatically be mapped to available devices. Thread 0 on slave 6 mapped to device 5 Thread 1 on slave 6 mapped to device 5 Thread 2 on slave 6 mapped to device 6 Thread 3 on slave 6 mapped to device 6 Thread 4 on slave 6 mapped to device 6 Thread 5 on slave 6 mapped to device 6 GPU-ids not specified for this rank, threads will automatically be mapped to available devices. Thread 0 on slave 7 mapped to device 6 Thread 1 on slave 7 mapped to device 7 Thread 2 on slave 7 mapped to device 7 Thread 3 on slave 7 mapped to device 7 Thread 4 on slave 7 mapped to device 7 Thread 5 on slave 7 mapped to device 7 Device 2 on gpu102 is split between 2 slaves Device 3 on gpu102 is split between 2 slaves Device 4 on gpu102 is split between 2 slaves Device 5 on gpu102 is split between 2 slaves Device 6 on gpu102 is split between 2 slaves Running CPU instructions in double precision. + WARNING: Changing psi sampling rate (before oversampling) to 11.25 degrees, for more efficient GPU calculations
On host gpu102: free scratch space = 532 Gb. Copying particles to scratch directory: /tmp/isomg01/relionvolatile/ 39.98/39.98 min ............................................................~~(,,"> Estimating initial noise spectra 29.83/29.83 min ............................................................~~(,,"> WARNING: There are only 3 particles in group 274 WARNING: There are only 1 particles in group 278 WARNING: There are only 2 particles in group 447 WARNING: There are only 1 particles in group 1242 WARNING: There are only 2 particles in group 1475 WARNING: There are only 1 particles in group 1506 WARNING: There are only 3 particles in group 1552 WARNING: There are only 2 particles in group 1736 WARNING: There are only 4 particles in group 1746 WARNING: There are only 1 particles in group 1778 WARNING: There are only 1 particles in group 1988 WARNING: There are only 3 particles in group 2431 WARNING: You may want to consider joining some micrographs into larger groups to obtain more robust noise estimates. You can do so by using the same rlnMicrographName for particles from multiple different micrographs in the input STAR file. It is then best to join micrographs with similar defocus values and similar apparent signal-to-noise ratios. Estimating accuracies in the orientational assignment ... 0/ 0 sec ............................................................~~(,,"> Auto-refine: Estimated accuracy angles= 999 degrees; offsets= 999 pixels CurrentResolution= 15.2269 Angstroms, which requires orientationSampling of at least 4.93151 degrees for a particle of diameter 350 Angstroms Oversampling= 0 NrHiddenVariableSamplingPoints= 134400 OrientationalSampling= 11.25 NrOrientations= 32 TranslationalSampling= 2 NrTranslations= 21
Oversampling= 1 NrHiddenVariableSamplingPoints= 4300800 OrientationalSampling= 5.625 NrOrientations= 256 TranslationalSampling= 1 NrTranslations= 84
Expectation iteration 1 of 50 36.87/37.22 hrs ...........................................................~~(,,"> Maximization ... 4/ 4 sec ............................................................~~(,,"> Estimating accuracies in the orientational assignment ... 0/ 0 sec ............................................................~~(,_,"> Auto-refine: Estimated accuracy angles= 999 degrees; offsets= 999 pixels CurrentResolution= 28.2786 Angstroms, which requires orientationSampling of at least 9.23077 degrees for a particle of diameter 350 Angstroms Oversampling= 0 NrHiddenVariableSamplingPoints= 134400 OrientationalSampling= 11.25 NrOrientations= 32 TranslationalSampling= 2 NrTranslations= 21
Oversampling= 1 NrHiddenVariableSamplingPoints= 4300800 OrientationalSampling= 5.625 NrOrientations= 256 TranslationalSampling= 1 NrTranslations= 84
Expectation iteration 2 of 50 32.51/32.82 hrs ...........................................................~~(,,"> Maximization ... 8/ 8 sec ............................................................~~(,,"> Estimating accuracies in the orientational assignment ... 0/ 0 sec ............................................................~~(,_,"> Auto-refine: Estimated accuracy angles= 16.1 degrees; offsets= 2.6 pixels CurrentResolution= 17.213 Angstroms, which requires orientationSampling of at least 5.625 degrees for a particle of diameter 350 Angstroms Oversampling= 0 NrHiddenVariableSamplingPoints= 134400 OrientationalSampling= 11.25 NrOrientations= 32 TranslationalSampling= 2 NrTranslations= 21
Oversampling= 1 NrHiddenVariableSamplingPoints= 4300800 OrientationalSampling= 5.625 NrOrientations= 256 TranslationalSampling= 1 NrTranslations= 84
Expectation iteration 3 of 50 34.61/34.94 hrs ...........................................................~~(,,"> Maximization ... 9/ 9 sec ............................................................~~(,,"> Estimating accuracies in the orientational assignment ... 3/ 3 sec ............................................................~~(,_,"> Auto-refine: Estimated accuracy angles= 3.1 degrees; offsets= 1.86 pixels CurrentResolution= 11.997 Angstroms, which requires orientationSampling of at least 3.91304 degrees for a particle of diameter 350 Angstroms Oversampling= 0 NrHiddenVariableSamplingPoints= 134400 OrientationalSampling= 11.25 NrOrientations= 32 TranslationalSampling= 2 NrTranslations= 21
Oversampling= 1 NrHiddenVariableSamplingPoints= 4300800 OrientationalSampling= 5.625 NrOrientations= 256 TranslationalSampling= 1 NrTranslations= 84
Expectation iteration 4 of 50 23.73/23.95 hrs ...........................................................~~(,,"> Maximization ... 4.77/4.77 min ............................................................~~(,,"> Estimating accuracies in the orientational assignment ... 5/ 5 sec ............................................................~~(,_,"> Auto-refine: Estimated accuracy angles= 1.5 degrees; offsets= 1.56 pixels CurrentResolution= 9.20698 Angstroms, which requires orientationSampling of at least 3 degrees for a particle of diameter 350 Angstroms Oversampling= 0 NrHiddenVariableSamplingPoints= 134400 OrientationalSampling= 11.25 NrOrientations= 32 TranslationalSampling= 2 NrTranslations= 21
Oversampling= 1 NrHiddenVariableSamplingPoints= 4300800 OrientationalSampling= 5.625 NrOrientations= 256 TranslationalSampling= 1 NrTranslations= 84
Expectation iteration 5 of 50 18.37/18.55 hrs ...........................................................~~(,,"> Maximization ... 5/ 5 sec ............................................................~~(,,"> Estimating accuracies in the orientational assignment ... 8/ 8 sec ............................................................~~(,_,"> Auto-refine: Estimated accuracy angles= 0.9 degrees; offsets= 0.97 pixels CurrentResolution= 7.46981 Angstroms, which requires orientationSampling of at least 2.43243 degrees for a particle of diameter 350 Angstroms Oversampling= 0 NrHiddenVariableSamplingPoints= 134400 OrientationalSampling= 11.25 NrOrientations= 32 TranslationalSampling= 2 NrTranslations= 21
Oversampling= 1 NrHiddenVariableSamplingPoints= 4300800 OrientationalSampling= 5.625 NrOrientations= 256 TranslationalSampling= 1 NrTranslations= 84
Expectation iteration 6 of 50 19.05/19.23 hrs ...........................................................~~(,,"> Maximization ... 6/ 6 sec ............................................................~~(,,"> Estimating accuracies in the orientational assignment ... 13/ 13 sec ............................................................~~(,_,"> Auto-refine: Estimated accuracy angles= 0.89 degrees; offsets= 1.26 pixels CurrentResolution= 6.28413 Angstroms, which requires orientationSampling of at least 2.05714 degrees for a particle of diameter 350 Angstroms Oversampling= 0 NrHiddenVariableSamplingPoints= 134400 OrientationalSampling= 11.25 NrOrientations= 32 TranslationalSampling= 2 NrTranslations= 21
Oversampling= 1 NrHiddenVariableSamplingPoints= 4300800 OrientationalSampling= 5.625 NrOrientations= 256 TranslationalSampling= 1 NrTranslations= 84
Expectation iteration 7 of 50 18.56/18.73 hrs ...........................................................~~(,,"> Maximization ... 34/ 34 sec ............................................................~~(,,"> Estimating accuracies in the orientational assignment ... 19/ 19 sec ............................................................~~(,_,"> Auto-refine: Estimated accuracy angles= 0.8 degrees; offsets= 1.15 pixels CurrentResolution= 5.42329 Angstroms, which requires orientationSampling of at least 1.7734 degrees for a particle of diameter 350 Angstroms Oversampling= 0 NrHiddenVariableSamplingPoints= 134400 OrientationalSampling= 11.25 NrOrientations= 32 TranslationalSampling= 2 NrTranslations= 21
Oversampling= 1 NrHiddenVariableSamplingPoints= 4300800 OrientationalSampling= 5.625 NrOrientations= 256 TranslationalSampling= 1 NrTranslations= 84
Expectation iteration 8 of 50 21.16/21.36 hrs ...........................................................~~(,,"> Maximization ... 1.25/1.25 min ............................................................~~(,,"> Estimating accuracies in the orientational assignment ... 23/ 23 sec ............................................................~~(,_,"> Auto-refine: Estimated accuracy angles= 0.8 degrees; offsets= 1.11 pixels CurrentResolution= 4.82805 Angstroms, which requires orientationSampling of at least 1.57895 degrees for a particle of diameter 350 Angstroms Oversampling= 0 NrHiddenVariableSamplingPoints= 134400 OrientationalSampling= 11.25 NrOrientations= 32 TranslationalSampling= 2 NrTranslations= 21
Oversampling= 1 NrHiddenVariableSamplingPoints= 4300800 OrientationalSampling= 5.625 NrOrientations= 256 TranslationalSampling= 1 NrTranslations= 84
Expectation iteration 9 of 50 22.84/23.05 hrs ...........................................................~~(,,"> Maximization ... 1.28/1.28 min ............................................................~~(,,"> Estimating accuracies in the orientational assignment ... 27/ 27 sec ............................................................~~(,_,"> Auto-refine: Estimated accuracy angles= 0.79 degrees; offsets= 1.07 pixels CurrentResolution= 4.76988 Angstroms, which requires orientationSampling of at least 1.55844 degrees for a particle of diameter 350 Angstroms Oversampling= 0 NrHiddenVariableSamplingPoints= 134400 OrientationalSampling= 11.25 NrOrientations= 32 TranslationalSampling= 2 NrTranslations= 21
Oversampling= 1 NrHiddenVariableSamplingPoints= 4300800 OrientationalSampling= 5.625 NrOrientations= 256 TranslationalSampling= 1 NrTranslations= 84
Expectation iteration 10 of 50 21.83/22.04 hrs ...........................................................~~(,,"> Maximization ... 7/ 7 sec ............................................................~~(,,"> Estimating accuracies in the orientational assignment ... 34/ 34 sec ............................................................~~(,_,"> Auto-refine: Estimated accuracy angles= 0.7 degrees; offsets= 1.06 pixels CurrentResolution= 4.60349 Angstroms, which requires orientationSampling of at least 1.50628 degrees for a particle of diameter 350 Angstroms Oversampling= 0 NrHiddenVariableSamplingPoints= 134400 OrientationalSampling= 11.25 NrOrientations= 32 TranslationalSampling= 2 NrTranslations= 21
Oversampling= 1 NrHiddenVariableSamplingPoints= 4300800 OrientationalSampling= 5.625 NrOrientations= 256 TranslationalSampling= 1 NrTranslations= 84
Expectation iteration 11 of 50 1.59/18.80 hrs .....~~(,_,">+++ RELION: command line arguments (with defaults for optional ones between parantheses) +++ ====== General options ===== --i : Input images (in a star-file or a stack) --o : Output rootname --iter (50) : Maximum number of iterations to perform --angpix (-1) : Pixel size (in Angstroms) --tau2_fudge (1) : Regularisation parameter (values higher than 1 give more weight to the data) --K (1) : Number of references to be refined --particle_diameter (-1) : Diameter of the circular mask that will be applied to the experimental images (in Angstroms) --zero_mask (false) : Mask surrounding background in particles to zero (by default the solvent area is filled with random noise) --flatten_solvent (false) : Perform masking on the references as well? --solvent_mask (None) : User-provided mask for the references (default is to use spherical mask with particle_diameter) --solvent_mask2 (None) : User-provided secondary mask (with its own average density) --multibody_masks () : STAR file with binary masks for multi-body refinement --tau (None) : STAR file with input tau2-spectrum (to be kept constant) --local_symmetry (None) : Local symmetry description file containing list of masks and their operators --split_random_halves (false) : Refine two random halves of the data completely separately --low_resol_join_halves (-1) : Resolution (in Angstrom) up to which the two random half-reconstructions will not be independent to prevent diverging orientations ====== Initialisation ===== --ref (None) : Image, stack or star-file with the reference(s). (Compulsory for 3D refinement!) --denovo_3dref (false) : Make an initial 3D model from randomly oriented 2D particles --offset (3) : Initial estimated stddev for the origin offsets --firstiter_cc (false) : Perform CC-calculation in the first iteration (use this if references are not on the absolute intensity scale) --ini_high (-1) : Resolution (in Angstroms) to which to limit refinement in the first iteration ====== Orientations ===== --oversampling (1) : Adaptive oversampling order to speed-up calculations (0=no oversampling, 1=2x, 2=4x, etc) --healpix_order (2) : Healpix order for the angular sampling (before oversampling) on the (3D) sphere: hp2=15deg, hp3=7.5deg, etc --psi_step (-1) : Sampling rate (before oversampling) for the in-plane angle (default=10deg for 2D, hp sampling for 3D) --limit_tilt (-91) : Limited tilt angle: positive for keeping side views, negative for keeping top views --sym (c1) : Symmetry group --offset_range (6) : Search range for origin offsets (in pixels) --offset_step (2) : Sampling rate (before oversampling) for origin offsets (in pixels) --helical_offset_step (-1) : Sampling rate (before oversampling) for offsets along helical axis (in pixels) --perturb (0.5) : Perturbation factor for the angular sampling (0=no perturb; 0.5=perturb) --auto_refine (false) : Perform 3D auto-refine procedure? --auto_local_healpix_order (4) : Minimum healpix order (before oversampling) from which autosampling procedure will use local searches --sigma_ang (-1) : Stddev on all three Euler angles for local angular searches (of +/- 3 stddev) --sigma_rot (-1) : Stddev on the first Euler angle for local angular searches (of +/- 3 stddev) --sigma_tilt (-1) : Stddev on the second Euler angle for local angular searches (of +/- 3 stddev) --sigma_psi (-1) : Stddev on the in-plane angle for local angular searches (of +/- 3 stddev) --skip_align (false) : Skip orientational assignment (only classify)? --skip_rotate (false) : Skip rotational assignment (only translate and classify)? --bimodal_psi (false) : Do bimodal searches of psi angle? ====== Helical reconstruction (in development...) ===== --helix (false) : Perform 3D classification or refinement for helices? --ignore_helical_symmetry (false) : Ignore helical symmetry? --helical_nr_asu (1) : Number of new helical asymmetric units (asu) per box (1 means no helical symmetry is present) --helical_twist_initial (0.) : Helical twist (in degrees, positive values for right-handedness) --helical_twist_min (0.) : Minimum helical twist (in degrees, positive values for right-handedness) --helical_twist_max (0.) : Maximum helical twist (in degrees, positive values for right-handedness) --helical_twist_inistep (0.) : Initial step of helical twist search (in degrees) --helical_rise_initial (0.) : Helical rise (in Angstroms) --helical_rise_min (0.) : Minimum helical rise (in Angstroms) --helical_rise_max (0.) : Maximum helical rise (in Angstroms) --helical_rise_inistep (0.) : Initial step of helical rise search (in Angstroms) --helical_z_percentage (0.3) : This box length along the center of Z axis contains good information of the helix. Important in imposing and refining symmetry --helical_inner_diameter (-1.) : Inner diameter of helical tubes in Angstroms (for masks of helical references and particles) --helical_outer_diameter (-1.) : Outer diameter of helical tubes in Angstroms (for masks of helical references and particles) --helical_symmetry_search (false) : Perform local refinement of helical symmetry? --helical_sigma_distance (-1.) : Sigma of distance along the helical tracks --helical_keep_tilt_prior_fixed (false) : Keep helical tilt priors fixed (at 90 degrees) in global angular searches? ====== Corrections ===== --ctf (false) : Perform CTF correction? --ctf_intact_first_peak (false) : Ignore CTFs until their first peak? --ctf_corrected_ref (false) : Have the input references been CTF-amplitude corrected? --ctf_phase_flipped (false) : Have the data been CTF phase-flipped? --ctf_multiplied (false) : Have the data been premultiplied with their CTF? --only_flip_phases (false) : Only perform CTF phase-flipping? (default is full amplitude-correction) --norm (false) : Perform normalisation-error correction? --scale (false) : Perform intensity-scale corrections on image groups? --no_norm (false) : Switch off normalisation-error correction? --no_scale (false) : Switch off intensity-scale corrections on image groups? ====== Stochastic Gradient Descent ===== --sgd (false) : Perform stochastic gradient descent instead of default expectation-maximization --mu (0.9) : Momentum parameter for SGD updates --subset_size (-1) : Size of the subsets for SGD --sgd_stepsize (0.5) : Step size parameter for SGD updates --max_subsets (-1) : Stop SGD after processing this many subsets (possibly more than 1 iteration) --sgd_sigma2fudge_initial (8) : Initial factor by which the noise variance will be multiplied for SGD (not used if halftime is negative) --sgd_sigma2fudge_halflife (-1) : Initialise SGD with 8x higher noise-variance, and reduce with this half-life in # of particles (default is keep normal variance) --write_subsets (-1) : Write out model every so many subsets (default is not writing any) --strict_highres_sgd (20) : Resolution limit (in Angstrom) to restrict probability calculations in SGD ====== Computation ===== --pool (1) : Number of images to pool for each thread task --j (1) : Number of threads to run in parallel (only useful on multi-core machines) --dont_combine_weights_via_disc (false) : Send the large arrays of summed weights through the MPI network, instead of writing large files to disc --onthefly_shifts (false) : Calculate shifted images on-the-fly, do not store precalculated ones in memory --no_parallel_disc_io (false) : Do NOT let parallel (MPI) processes access the disc simultaneously (use this option with NFS) --preread_images (false) : Use this to let the master process read all particles into memory. Be careful you have enough RAM for large data sets! --scratch_dir () : If provided, particle stacks will be copied to this local scratch disk prior to refinement. --keep_free_scratch (10) : Space available for copying particle stacks (in Gb) --reuse_scratch (false) : Re-use data on scratchdir, instead of wiping it and re-copying all data. --gpu (false) : Use available gpu resources for some calculations --free_gpu_memory (0) : GPU device memory (in Mb) to leave free after allocation. ====== Expert options ===== --pad (2) : Oversampling factor for the Fourier transforms of the references --NN (false) : Perform nearest-neighbour instead of linear Fourier-space interpolation? --r_min_nn (10) : Minimum number of Fourier shells to perform linear Fourier-space interpolation --verb (1) : Verbosity (1=normal, 0=silent) --random_seed (-1) : Number for the random seed generator --coarse_size (-1) : Maximum image size for the first pass of the adaptive sampling approach --adaptive_fraction (0.999) : Fraction of the weights to be considered in the first pass of adaptive oversampling --maskedge (5) : Width of the soft edge of the spherical mask (in pixels) --fix_sigma_noise (false) : Fix the experimental noise spectra? --fix_sigma_offset (false) : Fix the stddev in the origin offsets? --incr_size (10) : Number of Fourier shells beyond the current resolution to be included in refinement --print_metadata_labels (false) : Print a table with definitions of all metadata labels, and exit --print_symmetry_ops (false) : Print all symmetry transformation matrices, and exit --strict_highres_exp (-1) : Resolution limit (in Angstrom) to restrict probability calculations in the expectation step --dont_check_norm (false) : Skip the check whether the images are normalised correctly --always_cc (false) : Perform CC-calculation in all iterations (useful for faster denovo model generation?) --solvent_correct_fsc (false) : Correct FSC curve for the effects of the solvent mask? --skip_maximize (false) : Skip maximization step (only write out data.star file)? ====== MPI options ===== --only_do_unfinished_movies (false) : When processing movies on a per-micrograph basis, ignore those movies for which the output STAR file already exists.
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 1. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. --------------------------------------------------------------------------`
Please let me know if you know the origin of that error,
Thanks, Best, Nicolas