search

CAS-eResearch / TAOImport

TAO Data Conversion and Import Tools
GNU General Public License v3.0
2 stars 3 forks source link

MPI error while importing from tao #13

Closed manodeep closed 7 years ago

manodeep commented 7 years ago 
In [4]: from tao import Converter
--------------------------------------------------------------------------
Error obtaining unique transport key from ORTE (orte_precondition_transports not present in
the environment).

  Local host: g2
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  PML add procs failed
  --> Returned "Error" (-1) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[g2.hpc.swin.edu.au:21892] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed!
$ module list
    1) gmp/x86_64/gnu/5.0.5              4) isl/x86_64/gnu/0.14               7) openmpi/x86_64/gnu/1.6.1-psm     10) hdf5/x86_64/gnu/1.8.9-threadsafe 
    2) mpfr/x86_64/gnu/3.1.1             5) gcc/4.9.3                         8) gsl/x86_64/gnu/1.9               
    3) mpc/x86_64/gnu/1.0                6) intel/15.3.0                      9) gcc/4.7.1

ipython is from miniconda

Python 2.7.12 |Continuum Analytics, Inc.| (default, Jun 29 2016, 11:08:50) 
Type "copyright", "credits" or "license" for more information.

IPython 4.1.1 -- An enhanced Interactive Python.
manodeep commented 7 years ago

Solved by adding export OMPI_MCA_mtl=^psm before invoking python. But this error should not be happening.

manodeep commented 7 years ago

Something about the MPI environment on g2, particularly on the interactive nodes.