Memory leak with version 1.8.5?

[scrobey]

I'm seeing a large memory leak with version 1.8.5, installed with brew on Mac OSx 10.8.5. The same code does not have the problem with versions 1.8.1 to 1.8.4. This is just a heads up to see if others are seeing a similar problem.

[rhc54]

Can you tell us something about what you were doing to expose this so we can track it down?

[scrobey]

If you are interested in tracking it down, it is with the CLAMR open source adaptive mesh refinement code at http://www.github.com/losalamos/CLAMR. The particular problem that is failing is a hybrid MPI/GPU code. To reproduce it:

Download CLAMR source

https://github.com/losalamos/CLAMR.git cmake . make -j mpirun -n 2 ./clamr -n 128 -i 100 -t 600

Within a couple of hundred cycles, it has the 16 GB memory on my Mac laptop pegged and soon after that crashes the computer

Version 1.8.5 exhibits the problem, but 1.8.4 runs fine. I used the open-mpi 1.8.5 from brew that was released 3 days ago.

Strangely enough the mpionly version of the code runs without valgrind detecting any leaks

mpirun -n 2 ./clamr_mpionly -n 128 -i 100 -t 600

The leak seems to be at the MPI ghost cell updates which is a sparse all-to-all in the L7_Dev_Update/L7_Update calls.

On May 10, 2015, at 7:06 PM, rhc54 notifications@github.com wrote:

Can you tell us something about what you were doing to expose this so we can track it down?

— Reply to this email directly or view it on GitHub.

[ggouaillardet]

is a gpu needed to reproduce the issue ? I have no gpu and no clamr binary is built

[jsquyres]

Adding @rolfv, the NVIDIA Open MPI rep, in case it's a GPU issue...

[scrobey]

body{font-family: Geneva,Arial,Helvetica,sans-serif;font-size:9pt;background-color: #ffffff;color: black;}body{font-family: Geneva,Arial,Helvetica,sans-serif;font-size:9pt;background-color: #ffffff;color: black;}I think a GPU is necessary. The code path through the clamr_mpionly is almost identical in the MPI calls, but does not show the problem. Also, I am running GCC 4.9 and with NVidia GeForce GT 650M and an older version of OpenCL. Since this uses OpenCL, it is theoretically possible to run the OpenCL code on multi-core or other devices, but I have not tried that.-----Original Message----- From: Gilles Gouaillardet Sent: May 11, 2015 5:29 AM To: open-mpi/ompi Cc: Bob Subject: Re: [ompi] Memory leak with version 1.8.5? (#579)

is a gpu needed to reproduce the issue ? I have no gpu and no clamr binary is built

—Reply to this email directly or view it on GitHub.

[rolfv]

Do you what configure flags were used to configure Open MPI?

[scrobey]

body{font-family: Geneva,Arial,Helvetica,sans-serif;font-size:9pt;background-color: #ffffff;color: black;}I'm using the brew package for the Mac OSx. The website for their build is:https://github.com/Homebrew/homebrew/commit/931ef2827101198ea027a032f6e97d1f9d05b2cf#Library/Formula/open-mpi.rband the configuration formula is:class OpenMpi < Formula

      homepage "http://www.open-mpi.org/"

      url "http://www.open-mpi.org/software/ompi/v1.8/downloads/openmpi-1.8.5.tar.bz2"

      sha256 "4cea06a9eddfa718b09b8240d934b14ca71670c2dc6e6251a585ce948a93fbc4"

      bottle do

        sha1 "a6ec98d40ab34bf2eb4dbe9223d5aa430ba749ed" => :yosemite

        sha1 "9d7366e69787c6b331fe5473c8025d86d8b79691" => :mavericks

        sha1 "8c8627010c9390cb72054fba3f8eea419a67bb2b" => :mountain_lion

      end

      deprecated_option "disable-fortran" => "without-fortran"

      deprecated_option "enable-mpi-thread-multiple" => "with-mpi-thread-multiple"

      option "with-mpi-thread-multiple", "Enable MPI_THREAD_MULTIPLE"

      option :cxx11

      conflicts_with "mpich2", :because => "both install mpi__ compiler wrappers"

      conflicts_with "lcdf-typetools", :because => "both install same set of binaries."

      depends_on :fortran => :recommended

      depends_on "libevent"

      def install

        ENV.cxx11 if build.cxx11?

        args = %W[

          --prefix=#{prefix}

          --disable-dependency-tracking

          --disable-silent-rules

          --enable-ipv6

          --with-libevent=#{Formula["libevent"].opt_prefix}

        ]

        args << "--disable-mpi-fortran" if build.without? "fortran"

        args << "--enable-mpi-thread-multiple" if build.with? "mpi-thread-multiple"

        system "./configure", *args

        system "make", "all"

        system "make", "check"

        system "make", "install"

        # If Fortran bindings were built, there will be stray `.mod` files

        # (Fortran header) in `lib` that need to be moved to `include`.

        include.install Dir["#{lib}/*.mod"]

        # Move vtsetup.jar from bin to libexec.

        libexec.install bin/"vtsetup.jar"

        inreplace bin/"vtsetup", "$bindir/vtsetup.jar", "$prefix/libexec/vtsetup.jar"

      end

      def caveats; <<-EOS.undent

        WARNING: Open MPI now ignores the F77 and FFLAGS environment variables.

        Only the FC and FCFLAGS environment variables are used.

        EOS

      end

      test do

        (testpath/"hello.c").write <<-EOS.undent

          #include <mpi.h>

          #include <stdio.h>

          int main()

          {

            int size, rank, nameLen;

            char name[MPI_MAX_PROCESSOR_NAME];

            MPI_Init(NULL, NULL);

            MPI_Comm_size(MPI_COMM_WORLD, &size);

            MPI_Comm_rank(MPI_COMM_WORLD, &rank);

            MPI_Get_processor_name(name, &nameLen);

            printf("[%d/%d] Hello, world! My name is %s.\\n", rank, size, name);

            MPI_Finalize();

            return 0;

          }

        EOS

        system "#{bin}/mpicc", "hello.c", "-o", "hello"

        system "./hello"

        system "#{bin}/mpirun", "-np", "4", "./hello"

      end

    end-----Original Message-----

From: Rolf vandeVaart Sent: May 11, 2015 8:54 AM To: open-mpi/ompi Cc: Bob Subject: Re: [ompi] Memory leak with version 1.8.5? (#579)

Do you what configure flags were used to configure Open MPI?

—Reply to this email directly or view it on GitHub.

[jsquyres]

@scrobey Your last comment came through very garbled -- it looks like Github didn't understand the markup your mail client used.

Gilles and I made the assumption about GPU because you said it was an MPI/GPU code -- if you're not using GPUs, then that eliminates a whole bunch of variables.

[scrobey]

body{font-family: Geneva,Arial,Helvetica,sans-serif;font-size:9pt;background-color: #ffffff;color: black;}It may very well have something to do with the GPUs. I don't see the problem with the MPI only code despite the code paths being very similar through the MPI communication for the two versions. The configure instructions are from brew -- here is a link to the site rather than a cut and paste. The markdown must be messing with the formatting.https://github.com/Homebrew/homebrew/blob/931ef2827101198ea027a032f6e97d1f9d05b2cf/Library/Formula/open-mpi.rb-----Original Message----- From: Jeff Squyres Sent: May 11, 2015 10:38 AM To: open-mpi/ompi Cc: Bob Subject: Re: [ompi] Memory leak with version 1.8.5? (#579)

@scrobey Your last comment came through very garbled -- it looks like Github didn't understand the markup your mail client used.

Gilles and I made the assumption about GPU because you said it was an MPI/GPU code -- if you're not using GPUs, then that eliminates a whole bunch of variables.

—Reply to this email directly or view it on GitHub.

[rolfv]

I just ran this with Open MPI 1.8.5 and Open MPI 1.8.4 and they both ran fine for me. Here is 1.8.5 version.

[rvandevaart@drossetti-ivy4 CLAMR]$ which mpirun
/opt/openmpi/v1.8.5/bin/mpirun
[rvandevaart@drossetti-ivy4 CLAMR]$ 
[rvandevaart@drossetti-ivy4 CLAMR]$ mpirun -np 2 ./clamr -n 128 -t 600

Mass of initialized cells equal to  125195.039971
CPU:  setup time               time was   0.1322      0.1027    s
Memory used      in startup         105516      105448  kB
Memory peak      in startup         106392      106324  kB
Memory free      at startup       80868712    80868712  kB
Memory available at startup      132265516   132265516  kB
Iteration   0 timestep      n/a Sim Time      0.0 cells 17028 Mass Sum  125195.039971
Iteration 100 timestep 0.000330 Sim Time 0.043855 cells 17436 Mass Sum  125195.039971 Mass Change            0
Iteration 200 timestep 0.000318 Sim Time 0.076685 cells 17664 Mass Sum  125195.039971 Mass Change            0
Iteration 300 timestep 0.000346 Sim Time 0.109718 cells 17892 Mass Sum  125195.039971 Mass Change -1.45519e-11
Iteration 400 timestep 0.000358 Sim Time 0.144092 cells 18072 Mass Sum  125195.039971 Mass Change            0
Iteration 500 timestep 0.000371 Sim Time 0.180465 cells 18216 Mass Sum  125195.039971 Mass Change -1.45519e-11
Iteration 600 timestep 0.000398 Sim Time 0.219301 cells 18300 Mass Sum  125195.039971 Mass Change -1.45519e-11
Memory used             105792      105752  kB
Memory peak             106392      106324  kB
Memory free           80869476    80869476  kB
Memory available     132265516   132265516  kB

~~~~~~~~~~~~~~~~ Device timing information ~~~~~~~~~~~~~~~~~~

GPU: Parallel timings

GPU: Write to device          time was    0.0002      0.0003    s
GPU: Read from device         time was    0.0000      0.0000    s
GPU: Device compute           time was    2.4791      2.4783    s
GPU:   state_timer_set_timestep                 0.0977      0.0977  s
GPU:   state_timer_finite_difference            0.8978      0.8814  s
GPU:   state_timer_refine_potential             1.2578      1.2719  s
GPU:     state_timer_calc_mpot                    0.7152          0.7321    s
GPU:     mesh_timer_refine_smooth                 0.5425          0.5395    s
GPU:   mesh_timer_rezone_all                    0.0455      0.0455  s
GPU:   mesh_timer_partition                     0.0000      0.0000  s
GPU:   mesh_timer_calc_neighbors                0.1793      0.1809  s
GPU:     mesh_timer_hash_setup                    0.0267          0.0262    s
GPU:     mesh_timer_hash_query                    0.0092          0.0093    s
GPU:     mesh_timer_find_boundary                 0.0394          0.0364    s
GPU:     mesh_timer_push_setup                    0.0047          0.0083    s
GPU:     mesh_timer_push_boundary                 0.0018          0.0018    s
GPU:     mesh_timer_local_list                    0.0089          0.0083    s
GPU:     mesh_timer_layer1                        0.0098          0.0094    s
GPU:     mesh_timer_layer2                        0.0175          0.0176    s
GPU:     mesh_timer_layer_list                    0.0155          0.0157    s
GPU:     mesh_timer_copy_mesh_data                0.0142          0.0141    s
GPU:     mesh_timer_fill_mesh_ghost               0.0055          0.0057    s
GPU:     mesh_timer_fill_neigh_ghost              0.0091          0.0090    s
GPU:     mesh_timer_set_corner_neigh              0.0000          0.0000    s
GPU:     mesh_timer_neigh_adjust                  0.0083          0.0084    s
GPU:     mesh_timer_setup_comm                    0.0076          0.0097    s
GPU:   state_timer_mass_sum                     0.0010      0.0010  s
GPU:   mesh_timer_load_balance                  0.0000      0.0000  s
GPU:   mesh_timer_calc_spatial_coordi           0.0000      0.0000  s
=============================================================
Profiling: Total GPU          time was    2.4793      2.4786    s
-------------------------------------------------------------
Mesh Ops (Neigh+rezone+smooth+balance)    0.7673      0.7660    s
Mesh Ops Percentage                      30.9481     30.9024    percent
=============================================================
Profiling: Total              time was    2.4827      2.4827    s
=============================================================
CPU:  setup time               time was   0.0000      0.0000    s
GPU:  graphics                 time was   0.0000      0.0000    s
The MPI surface area to volume ratio      0.0323      0.0323    without duplicates
Using hash tables to calculate neighbors
Initial order is Hilbert sort.   No cycle reorder.   Local Stencil is on.
GPU:  rezone frequency                    12.1667   percent
GPU:  calc neigh frequency                12.3333   percent
GPU:  load balance frequency               0.0000   percent
GPU:  refine_smooth_iter per rezone        0.0000   
[drossetti-ivy4.nvidia.com:02867] 1 more process has sent help message help-mpi-btl-openib.txt / default subnet prefix
[drossetti-ivy4.nvidia.com:02867] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[rvandevaart@drossetti-ivy4 CLAMR]$ 

[scrobey]

I retested the Mac after a reboot and reinstalling openmpi 1.8.5 and it still leaks memory like crazy. I tried these options that Nathan Hjelm suggested. The first option did not get recognized and the second still has the memory leak. I can try some others and see if I can isolate which layer is causing the problem. I'll also try a 1.8.5 build on another system. None of our big systems have it installed yet.

The output below looks fine -- but it isn't wrong when the problem occurs. The memory usage just skyrockets. The memory usage in the run below looks OK, but top should be checked too. It may only be a Mac bug and this run below is not on a Mac?

Try these mpirun options and see if they help.

-mca oob usock -mca btl self,vader

-Nathan

On May 11, 2015, at 11:59 AM, Rolf vandeVaart notifications@github.com wrote:

I just ran this with Open MPI 1.8.5 and Open MPI 1.8.4 and they both ran fine for me. Here is 1.8.5 version.

[rvandevaart@drossetti-ivy4 CLAMR]$ which mpirun /opt/openmpi/v1.8.5/bin/mpirun [rvandevaart@drossetti-ivy4 CLAMR]$ [rvandevaart@drossetti-ivy4 CLAMR]$ mpirun -np 2 ./clamr -n 128 -t 600

Mass of initialized cells equal to 125195.039971 CPU: setup time time was 0.1322 0.1027 s Memory used in startup 105516 105448 kB Memory peak in startup 106392 106324 kB Memory free at startup 80868712 80868712 kB Memory available at startup 132265516 132265516 kB Iteration 0 timestep n/a Sim Time 0.0 cells 17028 Mass Sum 125195.039971 Iteration 100 timestep 0.000330 Sim Time 0.043855 cells 17436 Mass Sum 125195.039971 Mass Change 0 Iteration 200 timestep 0.000318 Sim Time 0.076685 cells 17664 Mass Sum 125195.039971 Mass Change 0 Iteration 300 timestep 0.000346 Sim Time 0.109718 cells 17892 Mass Sum 125195.039971 Mass Change -1.45519e-11 Iteration 400 timestep 0.000358 Sim Time 0.144092 cells 18072 Mass Sum 125195.039971 Mass Change 0 Iteration 500 timestep 0.000371 Sim Time 0.180465 cells 18216 Mass Sum 125195.039971 Mass Change -1.45519e-11 Iteration 600 timestep 0.000398 Sim Time 0.219301 cells 18300 Mass Sum 125195.039971 Mass Change -1.45519e-11 Memory used 105792 105752 kB Memory peak 106392 106324 kB Memory free 80869476 80869476 kB Memory available 132265516 132265516 kB

GPU: Parallel timings

GPU: Write to device          time was    0.0002      0.0003    s
GPU: Read from device         time was    0.0000      0.0000    s
GPU: Device compute           time was    2.4791      2.4783    s
GPU:   state_timer_set_timestep                 0.0977      0.0977  s
GPU:   state_timer_finite_difference            0.8978      0.8814  s
GPU:   state_timer_refine_potential             1.2578      1.2719  s
GPU:     state_timer_calc_mpot                    0.7152          0.7321    s
GPU:     mesh_timer_refine_smooth                 0.5425          0.5395    s
GPU:   mesh_timer_rezone_all                    0.0455      0.0455  s
GPU:   mesh_timer_partition                     0.0000      0.0000  s
GPU:   mesh_timer_calc_neighbors                0.1793      0.1809  s
GPU:     mesh_timer_hash_setup                    0.0267          0.0262    s
GPU:     mesh_timer_hash_query                    0.0092          0.0093    s
GPU:     mesh_timer_find_boundary                 0.0394          0.0364    s
GPU:     mesh_timer_push_setup                    0.0047          0.0083    s
GPU:     mesh_timer_push_boundary                 0.0018          0.0018    s
GPU:     mesh_timer_local_list                    0.0089          0.0083    s
GPU:     mesh_timer_layer1                        0.0098          0.0094    s
GPU:     mesh_timer_layer2                        0.0175          0.0176    s
GPU:     mesh_timer_layer_list                    0.0155          0.0157    s
GPU:     mesh_timer_copy_mesh_data                0.0142          0.0141    s
GPU:     mesh_timer_fill_mesh_ghost               0.0055          0.0057    s
GPU:     mesh_timer_fill_neigh_ghost              0.0091          0.0090    s
GPU:     mesh_timer_set_corner_neigh              0.0000          0.0000    s
GPU:     mesh_timer_neigh_adjust                  0.0083          0.0084    s
GPU:     mesh_timer_setup_comm                    0.0076          0.0097    s
GPU:   state_timer_mass_sum                     0.0010      0.0010  s
GPU:   mesh_timer_load_balance                  0.0000      0.0000  s

# GPU:   mesh_timer_calc_spatial_coordi           0.0000      0.0000  s

## Profiling: Total GPU          time was    2.4793      2.4786    s

Mesh Ops (Neigh+rezone+smooth+balance)    0.7673      0.7660    s

# Mesh Ops Percentage                      30.9481     30.9024    percent

# Profiling: Total              time was    2.4827      2.4827    s

CPU:  setup time               time was   0.0000      0.0000    s
GPU:  graphics                 time was   0.0000      0.0000    s
The MPI surface area to volume ratio      0.0323      0.0323    without duplicates
Using hash tables to calculate neighbors
Initial order is Hilbert sort.   No cycle reorder.   Local Stencil is on.
GPU:  rezone frequency                    12.1667   percent
GPU:  calc neigh frequency                12.3333   percent
GPU:  load balance frequency               0.0000   percent
GPU:  refine_smooth_iter per rezone        0.0000  
[drossetti-ivy4.nvidia.com:02867] 1 more process has sent help message help-mpi-btl-openib.txt / default subnet prefix
[drossetti-ivy4.nvidia.com:02867] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[rvandevaart@drossetti-ivy4 CLAMR]$ 
—
Reply to this email directly or view it on GitHub.

[scrobey]

I just looked at top on the Mac when it is leaking memory and it is orterun that is showing the large memory use rather than the 2 clamr processes. Does this tell us anything? Top at around 80 cycles

8885 clamr 64.5 00:27.53 8/1 5 71 393+ 9484K+ 26M+ 20M+ 79M+ 2675M+ 8885 8882 running 8884 clamr 36.6 00:24.90 7 4 71+ 393+ 9620K- 27M+ 20M 94M- 2675M+ 8884 8882 sleeping 8882 orterun 77.0 00:41.07 2/1 0 22 13387+ 13G+ 1224K 13G- 13G+ 15G+ 8882 1255 running

On May 11, 2015, at 11:59 AM, Rolf vandeVaart notifications@github.com wrote:

I just ran this with Open MPI 1.8.5 and Open MPI 1.8.4 and they both ran fine for me. Here is 1.8.5 version.

[rvandevaart@drossetti-ivy4 CLAMR]$ which mpirun /opt/openmpi/v1.8.5/bin/mpirun [rvandevaart@drossetti-ivy4 CLAMR]$ [rvandevaart@drossetti-ivy4 CLAMR]$ mpirun -np 2 ./clamr -n 128 -t 600

Mass of initialized cells equal to 125195.039971 CPU: setup time time was 0.1322 0.1027 s Memory used in startup 105516 105448 kB Memory peak in startup 106392 106324 kB Memory free at startup 80868712 80868712 kB Memory available at startup 132265516 132265516 kB Iteration 0 timestep n/a Sim Time 0.0 cells 17028 Mass Sum 125195.039971 Iteration 100 timestep 0.000330 Sim Time 0.043855 cells 17436 Mass Sum 125195.039971 Mass Change 0 Iteration 200 timestep 0.000318 Sim Time 0.076685 cells 17664 Mass Sum 125195.039971 Mass Change 0 Iteration 300 timestep 0.000346 Sim Time 0.109718 cells 17892 Mass Sum 125195.039971 Mass Change -1.45519e-11 Iteration 400 timestep 0.000358 Sim Time 0.144092 cells 18072 Mass Sum 125195.039971 Mass Change 0 Iteration 500 timestep 0.000371 Sim Time 0.180465 cells 18216 Mass Sum 125195.039971 Mass Change -1.45519e-11 Iteration 600 timestep 0.000398 Sim Time 0.219301 cells 18300 Mass Sum 125195.039971 Mass Change -1.45519e-11 Memory used 105792 105752 kB Memory peak 106392 106324 kB Memory free 80869476 80869476 kB Memory available 132265516 132265516 kB

GPU: Parallel timings

GPU: Write to device          time was    0.0002      0.0003    s
GPU: Read from device         time was    0.0000      0.0000    s
GPU: Device compute           time was    2.4791      2.4783    s
GPU:   state_timer_set_timestep                 0.0977      0.0977  s
GPU:   state_timer_finite_difference            0.8978      0.8814  s
GPU:   state_timer_refine_potential             1.2578      1.2719  s
GPU:     state_timer_calc_mpot                    0.7152          0.7321    s
GPU:     mesh_timer_refine_smooth                 0.5425          0.5395    s
GPU:   mesh_timer_rezone_all                    0.0455      0.0455  s
GPU:   mesh_timer_partition                     0.0000      0.0000  s
GPU:   mesh_timer_calc_neighbors                0.1793      0.1809  s
GPU:     mesh_timer_hash_setup                    0.0267          0.0262    s
GPU:     mesh_timer_hash_query                    0.0092          0.0093    s
GPU:     mesh_timer_find_boundary                 0.0394          0.0364    s
GPU:     mesh_timer_push_setup                    0.0047          0.0083    s
GPU:     mesh_timer_push_boundary                 0.0018          0.0018    s
GPU:     mesh_timer_local_list                    0.0089          0.0083    s
GPU:     mesh_timer_layer1                        0.0098          0.0094    s
GPU:     mesh_timer_layer2                        0.0175          0.0176    s
GPU:     mesh_timer_layer_list                    0.0155          0.0157    s
GPU:     mesh_timer_copy_mesh_data                0.0142          0.0141    s
GPU:     mesh_timer_fill_mesh_ghost               0.0055          0.0057    s
GPU:     mesh_timer_fill_neigh_ghost              0.0091          0.0090    s
GPU:     mesh_timer_set_corner_neigh              0.0000          0.0000    s
GPU:     mesh_timer_neigh_adjust                  0.0083          0.0084    s
GPU:     mesh_timer_setup_comm                    0.0076          0.0097    s
GPU:   state_timer_mass_sum                     0.0010      0.0010  s
GPU:   mesh_timer_load_balance                  0.0000      0.0000  s

# GPU:   mesh_timer_calc_spatial_coordi           0.0000      0.0000  s

## Profiling: Total GPU          time was    2.4793      2.4786    s

Mesh Ops (Neigh+rezone+smooth+balance)    0.7673      0.7660    s

# Mesh Ops Percentage                      30.9481     30.9024    percent

# Profiling: Total              time was    2.4827      2.4827    s

CPU:  setup time               time was   0.0000      0.0000    s
GPU:  graphics                 time was   0.0000      0.0000    s
The MPI surface area to volume ratio      0.0323      0.0323    without duplicates
Using hash tables to calculate neighbors
Initial order is Hilbert sort.   No cycle reorder.   Local Stencil is on.
GPU:  rezone frequency                    12.1667   percent
GPU:  calc neigh frequency                12.3333   percent
GPU:  load balance frequency               0.0000   percent
GPU:  refine_smooth_iter per rezone        0.0000  
[drossetti-ivy4.nvidia.com:02867] 1 more process has sent help message help-mpi-btl-openib.txt / default subnet prefix
[drossetti-ivy4.nvidia.com:02867] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[rvandevaart@drossetti-ivy4 CLAMR]$ 
—
Reply to this email directly or view it on GitHub.

[rolfv]

That is perplexing. I do not see that on my machine but it is RHEL 6.5. For the record, here is some output from top just as the job gets going. It will be interesting to see if you observe it on another machine.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                               
30809 rvandeva  20   0  164g 103m  83m R 51.9  0.1   0:01.56 clamr                                 
30810 rvandeva  20   0  164g 103m  83m R 51.9  0.1   0:01.56 clamr                                 
30807 rvandeva  20   0  158m 3500 2432 S  1.0  0.0   0:00.03 mpirun               

[scrobey]

body{font-family: Geneva,Arial,Helvetica,sans-serif;font-size:9pt;background-color: #ffffff;color: black;}I'll also rebuild open-mpi 1.8.5 from scratch on the Mac and maybe a newer version of the Mac OS. -----Original Message----- From: Rolf vandeVaart Sent: May 12, 2015 7:05 AM To: open-mpi/ompi Cc: Bob Subject: Re: [ompi] Memory leak with version 1.8.5? (#579)

That is perplexing. I do not see that on my machine but it is RHEL 6.5. For the record, here is some output from top just as the job gets going. It will be interesting to see if you observe it on another machine.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30809 rvandeva 20 0 164g 103m 83m R 51.9 0.1 0:01.56 clamr
30810 rvandeva 20 0 164g 103m 83m R 51.9 0.1 0:01.56 clamr
30807 rvandeva 20 0 158m 3500 2432 S 1.0 0.0 0:00.03 mpirun

—Reply to this email directly or view it on GitHub.

[ianhinder]

I am seeing the same problem with the Cactus code (cactuscode.org) also with OpenMPI 1.8.5 installed from homebrew. The orterun process takes a huge amount of memory, whereas the MPI process memory usage is normal (about 4 GB, as intended). This is with -np 1.

[scrobey]

What is the platform you are seeing this on? Is it just MPI code or is it a hybrid MPI/+X code?

On May 12, 2015, at 9:39 AM, ianhinder notifications@github.com wrote:

I am seeing the same problem with the Cactus code (cactuscode.org) also with OpenMPI 1.8.5 installed from homebrew. The orterun process takes a huge amount of memory, whereas the MPI process memory usage is normal (about 4 GB, as intended). This is with -np 1.

— Reply to this email directly or view it on GitHub.

[ianhinder]

We see the problem on OS 10.9 and OS 10.10. In both instances, OpenMPI was installed via Homebrew (https://github.com/Homebrew/homebrew/blob/master/Library/Formula/open-mpi.rb). The homebrew commit (https://github.com/Homebrew/homebrew/commit/931ef2827101198ea027a032f6e97d1f9d05b2cf) which updated 1.8.4 to 1.8.5 does not seem to make any other changes, so I suspect this is not a homebrew problem.

It is MPI code with OpenMP, and I am running on a single process with 2 OpenMP threads. We have reverted to using OpenMP 1.6, because there is a homebrew package for that version. Unfortunately I am not able to test with 1 thread right now; if needed I can do this in the next few days.

[ggouaillardet]

@ianhinder can you please provide basic instructions on how to build and run cactuscode ? is some input data required ? if yes, do they come with the code ? if not can you make them available ? fwiw, I note homebrew configure with --enable-ipv6

[rhc54]

I can reproduce this on the Mac - don't see it on Linux. I'm not sure of the root cause but will investigate.

FWIW: it seems to be happening regardless of configuration, and even with just "sleep 1000" as the application. The key is to keep mpirun alive for awhile. Everything is stable for roughly 30 seconds, and then something triggers a rapid growth in memory footprint accompanied by 100% cpu.

[ianhinder]

To reproduce this using Cactus (I wish it were simpler):

brew tap homebrew/science
brew install curl gcc --without-multilib fftw gsl hdf5 --with-fortran hwloc jpeg openssl pkg-config szip open-mpi

Apple's gcc is not enough, which is why you need the version from homebrew.

Apple's SVN doesn't like the Cactus server certificate. Either use an svn which has a modern certificate chain, or do

svn list https://svn.cactuscode.org/projects/ExternalLibraries/pciutils
svn list https://svn.aei.mpg.de/numrel/AEIThorns/SystemStatistics

and accept the certificates permanently.

curl -kLO https://raw.githubusercontent.com/gridaphobe/CRL/master/GetComponents
chmod a+x GetComponents
./GetComponents --parallel https://gist.githubusercontent.com/ianhinder/62dd8f2cf3da340364d9/raw/0a6a3e35b0771381931ecec4d9bd2bda74ee353a/cactus.th

cd Cactus

Edit simfactory/mdb/optionlists/os-homebrew.cfg and change MPI_DIR to /usr/local.

make sim-config options=simfactory/mdb/optionlists/osx-homebrew.cfg THORNLIST=../cactus.th
make

Now run Cactus with the memspeed test:

/usr/local/bin/mpirun -np 1 exe/cactus_sim arrangements/CactusUtils/MemSpeed/test/memspeed.par

Watch orterun grow to many GB in activity monitor.

[rhc54]

As I said, it is unnecessary to go to those lengths - a simple "sleep 10000" will do. Still investigating. I'm not finding anything obvious, so this may take a bit. Clearly something limited to the Mac.

[ianhinder]

And the last hour was wasted...

[rhc54]

Sorry - I did get to it as soon as I could :-(

I have confirmed that this isn't happening on the master. Am now checking with earlier versions in the 1.8 series.

[rhc54]

I've confirmed that this does not exist in 1.8.4. Still no clear root cause, but at least there is an interim solution - just stick with 1.8.4 until we figure this out.

[scrobey]

body{font-family: Geneva,Arial,Helvetica,sans-serif;font-size:9pt;background-color: #ffffff;color: black;}I rebuilt 1.8.5 on the Mac with a simple configure (only a prefix option) and it still reproduces the problem. Also running with mpirun -n 1 shows a leak though it doesn't seem to be as bad. The description using sleep matches what I see. It seems OK for some seconds and then starts leaking like crazy.-----Original Message----- From: Gilles Gouaillardet Sent: May 13, 2015 6:12 AM To: open-mpi/ompi Cc: Bob Subject: Re: [ompi] Memory leak with version 1.8.5? (#579)

@ianhinder can you please provide basic instructions on how to build and run cactuscode ? is some input data required ? if yes, do they come with the code ? if not can you make them available ? fwiw, I note homebrew configure with --enable-ipv6

—Reply to this email directly or view it on GitHub.

[barrywardell]

As of https://github.com/Homebrew/homebrew/pull/39745, homebrew has reverted to using 1.8.4 until a fix is found.

[rhc54]

I have this fixed and awaiting commit into the 1.8 series:

https://github.com/open-mpi/ompi-release/pull/275

[scrobey]

I tried out the fixes and now there is no problem with running on the Mac. Nice work.

Bob On May 14, 2015, at 6:18 PM, rhc54 notifications@github.com wrote:

I have this fixed and awaiting commit into the 1.8 series:

open-mpi/ompi-release#275

— Reply to this email directly or view it on GitHub.

[jsquyres]

@barrywardell FYI: You can still stick with 1.8.5 (there have been many bug fixes since 1.8.4) and just disable the "keepalive" functionality by adding this line to the openmpi-mca-params.conf file that is installed in $(sysconfdir):

oob_tcp_keepalive_time = 0

[jsquyres]

This is now fixed on master and the v1.8 branch -- will be in the upcoming v1.8.6 release.

[scrobey]

Thanks for the quick response on this. Does this mean that we should not install 1.8.5 on Macs? Or is the fix being back-ported?

Bob On May 22, 2015, at 2:34 PM, Jeff Squyres notifications@github.com wrote:

This is now fixed on master and the v1.8 branch -- will be in the upcoming v1.8.6 release.

— Reply to this email directly or view it on GitHub.

[rhc54]

You can use 1.8.5 on Mac, but you need to set the MCA param oob_tcp_keepalive_time = 0 in your default MCA param file to ensure it is always applied

FWIW: I plan to release 1.8.6 (which will include the fix) in about 1 week

[RyanMarcus]

Just wanted to note that Homebrew upgraded to 1.8.6 around July 6th, so if anyone stumbles into this issue, a quick "brew update && brew upgrade" should fix the problem.

[jsquyres]

Sweet -- thanks for the heads up.