Portals 4 Implementation Notes

There is currently one Portals 4 implementation, ib. The ib implementation supports multiple transports, including InfiniBand, UDP/Reliable UDP (experimental), and SHMEM.

• Infiniband: this transport has multiple options, including not actualling using IB:

  • Infiniband, selected by default, can be disabled with --disable-transport-ib

  • shared memory, not selected by default, can be enabled with --enable-transport-shmem. It has the option of using KNEM for the large transfer (used if --with-knem=xxxx is present), or to use an internal slower shared memory protocol.

  Both Infiniband and shmem can be used at the same time. In that case, Portals will use infinband between nodes, and shmem intra-node. If only shmem is used, then portals will not run between nodes (obviously).

On top of that, 2 different environments can be generated:

• the "fat library" which contains all the portals API. Each process has its own progress thread which leads to the use of too many cores on a machine. This is the default.

• the Portals Progres Engine (PPE) and the light library. The PPE is a daemon that regroups the progress threads (one by default) for all the portals process on the host. The portals application links to the "light library" which is only a pass through between the application and the PPE. Very litlle processing is done in the light library; most of it is done inside the PPE. This environment is selected with --enable-ib-ppe. The XPMEM driver must be present, and so it is mutually exclusive with the shmem transport. The daemon must be started prior to running a Portals4 application:

  /p4ppe p4ppe has a set of options. Run with the --help optins to see them.

Both the fat and light library are called the same, thus only one type can exist on a machine at the same time; use LD_LIBRARY_PATH and/or LD_PRELOAD to work around.

Pre-requisites

• libev-4.0.4 (or similar), available at http://software.schmorp.de/pkg/libev.html

• The KNEM driver from http://runtime.bordeaux.inria.fr/knem/. Will be used for shared memory if --with-knem= is set. Ensure that /dev/knem is readable/writable by the user running the portals software.

• The XPMEM driver from https://code.google.com/p/xpmem/. Mandatory if the PPE is selected, unused otherwise. The current driver has crashing issues and doesn't compile on recent kernel versions.

• The ummunotify driver from https://github.com/Portals4/ummunotify/ (required for correctness with the IB transport). Ensure that /dev/ummunotify is readable/writable by the user running the portals software. WARNING: Not using ummunotify may result in program incorrectness..

Build: Note that the paths to mpi may not be necessary or may need to be changed depending on your linux distribution.

• minimal, no shared memory: ./configure

• optimized, with shared memory and KNEM support: ./configure --enable-transport-shmem --with-knem=/opt/knem --enable-fast

• PPE, no IB transport (so local node only): ./configure --enable-ib-ppe --disable-transport-ib

  Then type "make".

Test:

• Type "make check" in the main directory. This will run a series of checks on the local node, with either 1 or 2 ranks. Configure will attempt to figure out the proper way to launch parallel processes. The TEST_RUNNER environment variable can be used to set the proper test runner (such as TEST_RUNNER='yod -np $(NPROCS)').

Environment variables:

• PTL_ENABLE_MEM=[0|1] will deactivate/activate the local memory transport, if one is compiled in.
• PTL_DEBUG=1 will activate the tracing
• PTL_LOG_LEVEL=[0|1|2|3] will set the trace level.
• PTL_IFACE_NAME allows for the explicit naming of the network interface for example ib0, eno1, etc.

• PTL_DISABLE_MEM_REG_CACHE=[0|1] deactivates/activates the IB memory registration cache. Disabling it no longer requires ummunotify, and the implementation does not keep a registered memory cache.

  For instance: PTL_LOG_LEVEL=3 PTL_DEBUG=1 yod -n 1 ./spam

Building an RPM:

• make dist --> this will create portals-4.0.tar.gz
• rpmbuild -ta portals-4.0.tar.gz

Limitations:

• A program may not open more than one NID in its lifetime.

Why Portals may not run properly:

• The user cannot pin enough memory. By default linux systems allow about 32KB. Check with "ulimit -l". This is for IB only. Edit /etc/security/limits.co onf and change/add the following 2 lines (including the stars):

  • hard memlock unlimited
  • soft memlock unlimited (This setting allow every user to pin large amounts of memory. Replace the stars with a user name for a tighter setting).

• IB is not up and running (check with "ibv_devinfo") or the default IPoIB interface, ib0, doesn't have an IP address. All interfaces should be on the same IP range.

• slow performances can be caused by cpuspeed. Disable it ("service cpuspeed stop").

• uneven performance between repeats can be caused by the threads being on different cores. Pin them.

• When shmem is used, under the wrong circumstances, the files /dev/shm/portals4-shmem-* are not removed. If a subsequent run tries to re-use the same name, it will fail and the application may exit.

• If you get a mr_lookup: Assertion `res == ((void *)0)' failed error, you need to configure Portals 4 with --enable-zero-mrs to enable zero length MR suport.