Backfilling scheduler backfills erratically on 3 pilots

[mturilli]

Running a 1024 BoT on stampede, trestles, and gordon. Session UID: 5402008c20a6417570206001

Observed behavior: trestles pilot comes online almost immediately. 341 CUs are executed and no stage out happens. RP waits until gordon comes up, it executes 480 CUs (?!), all fail (expected due to the mpi issue). RP waits for stampede to come up, 203 CUs are executed and they successfully execute.

Notes:

• 1015 files are successfully staged out (203 Units, consistent with the behavior observed).

• The 341 units executed on trestles shows the same/similar error reported at #309 on gordon:

  Unloading compiler-dependent module mvapich2_ib/1.9
  loading intel and mvapich2_ib modules
                     :-)  G  R  O  M  A  C  S  (-:
  
                  GROwing Monsters And Cloning Shrimps
  
                        :-)  VERSION 4.6.5  (-:
  
    Contributions from Mark Abraham, Emile Apol, Rossen Apostolov, 
       Herman J.C. Berendsen, Aldert van Buuren, Pär Bjelkmar,  
  Rudi van Drunen, Anton Feenstra, Gerrit Groenhof, Christoph Junghans, 
    Peter Kasson, Carsten Kutzner, Per Larsson, Pieter Meulenhoff, 
       Teemu Murtola, Szilard Pall, Sander Pronk, Roland Schulz, 
            Michael Shirts, Alfons Sijbers, Peter Tieleman,
  
           Berk Hess, David van der Spoel, and Erik Lindahl.
  
   Copyright (c) 1991-2000, University of Groningen, The Netherlands.
     Copyright (c) 2001-2012,2013, The GROMACS development team at
    Uppsala University & The Royal Institute of Technology, Sweden.
        check out http://www.gromacs.org for more information.
  
     This program is free software; you can redistribute it and/or
   modify it under the terms of the GNU Lesser General Public License
    as published by the Free Software Foundation; either version 2.1
         of the License, or (at your option) any later version.
  
                          :-)  mdrun_mpi  (-:
  
  Option     Filename  Type         Description
  ------------------------------------------------------------
  -s      topol.tpr  Input        Run input file: tpr tpb tpa
  -o       traj.trr  Output       Full precision trajectory: trr trj cpt
  -x       traj.xtc  Output, Opt. Compressed trajectory (portable xdr format)
  -cpi      state.cpt  Input, Opt.  Checkpoint file
  -cpo      state.cpt  Output, Opt. Checkpoint file
  -c    confout.gro  Output       Structure file: gro g96 pdb etc.
  -e       ener.edr  Output       Energy file
  -g         md.log  Output       Log file
  -dhdl      dhdl.xvg  Output, Opt. xvgr/xmgr file
  -field    field.xvg  Output, Opt. xvgr/xmgr file
  -table    table.xvg  Input, Opt.  xvgr/xmgr file
  -tabletf    tabletf.xvg  Input, Opt.  xvgr/xmgr file
  -tablep  tablep.xvg  Input, Opt.  xvgr/xmgr file
  -tableb   table.xvg  Input, Opt.  xvgr/xmgr file
  -rerun    rerun.xtc  Input, Opt.  Trajectory: xtc trr trj gro g96 pdb cpt
  -tpi        tpi.xvg  Output, Opt. xvgr/xmgr file
  -tpid   tpidist.xvg  Output, Opt. xvgr/xmgr file
  -ei        sam.edi  Input, Opt.  ED sampling input
  -eo      edsam.xvg  Output, Opt. xvgr/xmgr file
  -j       wham.gct  Input, Opt.  General coupling stuff
  -jo        bam.gct  Output, Opt. General coupling stuff
  -ffout      gct.xvg  Output, Opt. xvgr/xmgr file
  -devout   deviatie.xvg  Output, Opt. xvgr/xmgr file
  -runav  runaver.xvg  Output, Opt. xvgr/xmgr file
  -px      pullx.xvg  Output, Opt. xvgr/xmgr file
  -pf      pullf.xvg  Output, Opt. xvgr/xmgr file
  -ro   rotation.xvg  Output, Opt. xvgr/xmgr file
  -ra  rotangles.log  Output, Opt. Log file
  -rs   rotslabs.log  Output, Opt. Log file
  -rt  rottorque.log  Output, Opt. Log file
  -mtx         nm.mtx  Output, Opt. Hessian matrix
  -dn     dipole.ndx  Output, Opt. Index file
  -multidir    rundir  Input, Opt., Mult. Run directory
  -membed  membed.dat  Input, Opt.  Generic data file
  -mp     membed.top  Input, Opt.  Topology file
  -mn     membed.ndx  Input, Opt.  Index file
  
  Option       Type   Value   Description
  ------------------------------------------------------
  -[no]h       bool   no      Print help info and quit
  -[no]version bool   no      Print version info and quit
  -nice        int    0       Set the nicelevel
  -deffnm      string         Set the default filename for all file options
  -xvg         enum   xmgrace  xvg plot formatting: xmgrace, xmgr or none
  -[no]pd      bool   no      Use particle decompostion
  -dd          vector 0 0 0   Domain decomposition grid, 0 is optimize
  -ddorder     enum   interleave  DD node order: interleave, pp_pme or cartesian
  -npme        int    -1      Number of separate nodes to be used for PME, -1
                          is guess
  -nt          int    0       Total number of threads to start (0 is guess)
  -ntmpi       int    0       Number of thread-MPI threads to start (0 is guess)
  -ntomp       int    0       Number of OpenMP threads per MPI process/thread
                          to start (0 is guess)
  -ntomp_pme   int    0       Number of OpenMP threads per MPI process/thread
                          to start (0 is -ntomp)
  -pin         enum   auto    Fix threads (or processes) to specific cores:
                          auto, on or off
  -pinoffset   int    0       The starting logical core number for pinning to
                          cores; used to avoid pinning threads from
                          different mdrun instances to the same core
  -pinstride   int    0       Pinning distance in logical cores for threads,
                          use 0 to minimize the number of threads per
                          physical core
  -gpu_id      string         List of GPU device id-s to use, specifies the
                          per-node PP rank to GPU mapping
  -[no]ddcheck bool   yes     Check for all bonded interactions with DD
  -rdd         real   0       The maximum distance for bonded interactions with
                          DD (nm), 0 is determine from initial coordinates
  -rcon        real   0       Maximum distance for P-LINCS (nm), 0 is estimate
  -dlb         enum   auto    Dynamic load balancing (with DD): auto, no or yes
  -dds         real   0.8     Minimum allowed dlb scaling of the DD cell size
  -gcom        int    -1      Global communication frequency
  -nb          enum   auto    Calculate non-bonded interactions on: auto, cpu,
                          gpu or gpu_cpu
  -[no]tunepme bool   yes     Optimize PME load between PP/PME nodes or GPU/CPU
  -[no]testverlet bool   no      Test the Verlet non-bonded scheme
  -[no]v       bool   no      Be loud and noisy
  -[no]compact bool   yes     Write a compact log file
  -[no]seppot  bool   no      Write separate V and dVdl terms for each
                          interaction type and node to the log file(s)
  -pforce      real   -1      Print all forces larger than this (kJ/mol nm)
  -[no]reprod  bool   no      Try to avoid optimizations that affect binary
                          reproducibility
  -cpt         real   15      Checkpoint interval (minutes)
  -[no]cpnum   bool   no      Keep and number checkpoint files
  -[no]append  bool   yes     Append to previous output files when continuing
                          from checkpoint instead of adding the simulation
                          part number to all file names
  -nsteps      step   -2      Run this number of steps, overrides .mdp file
                          option
  -maxh        real   -1      Terminate after 0.99 times this time (hours)
  -multi       int    0       Do multiple simulations in parallel
  -replex      int    0       Attempt replica exchange periodically with this
                          period (steps)
  -nex         int    0       Number of random exchanges to carry out each
                          exchange interval (N^3 is one suggestion).  -nex
                          zero or not specified gives neighbor replica
                          exchange.
  -reseed      int    -1      Seed for replica exchange, -1 is generate a seed
  -[no]ionize  bool   no      Do a simulation including the effect of an X-Ray
                          bombardment on your system
  
  Reading file topol.tpr, VERSION 4.6.5 (single precision)
  Using 1 MPI process
  
  Non-default thread affinity set probably by the OpenMP library,
  disabling internal thread affinity
  starting mdrun 'Protein t=   0.00000'
  100000 steps,    100.0 ps.
  
  Writing final coordinates.
  
             Core t (s)   Wall t (s)        (%)
     Time:        7.430     2233.825        0.3
                       37:13
               (ns/day)    (hour/ns)
  Performance:        3.868        6.205
  
  gcq#134: "Catholic School Girls Rule" (Red Hot Chili Peppers)
  
  [mpiexec@trestles-3-31.sdsc.edu] HYDT_bscd_pbs_wait_for_completion         (./tools/bootstrap/external/pbs_wait.c:68): tm_poll(obit_event) failed with TM error 17002
  [mpiexec@trestles-3-31.sdsc.edu] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion
  [mpiexec@trestles-3-31.sdsc.edu] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217):   launcher returned error waiting for completion
  [mpiexec@trestles-3-31.sdsc.edu] main (./ui/mpich/mpiexec.c:331): process manager error waiting for   completion

• The error on gordon's CU:

  Received the TERM signal, stopping at the next NS step
  
  [gcn-14-71.sdsc.edu:mpi_rank_0][error_sighandler] Caught error: Aborted (signal 6)
  [mpiexec@gcn-14-71.sdsc.edu] HYDT_bscd_pbs_wait_for_completion   (./tools/bootstrap/external/pbs_wait.c:48): tm_obit() failed with TM error 17000
  [mpiexec@gcn-14-71.sdsc.edu] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23):   launcher returned error waiting for completion
  [mpiexec@gcn-14-71.sdsc.edu] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217):   launcher returned error waiting for completion
  [mpiexec@gcn-14-71.sdsc.edu] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion

  

[andre-merzky]

Actually, I think the scheduler does ok -- the pilots are not looking good, nor does the CU execution.

From the perspective of the BF scheduler, at ~200s the trestles pilot becomes available, and gets its share of units assigned. As those units never complete in the pilot lifetime, no new units get scheduled in that time. At ~3000s, the gordon pilot becomes active, and also gets its share of units. Again, no unit seems to make it through to a final state, they seem to hang in execution. At ~4300 (and this is hard to see, but you'll find it if you look closely), some of Gordon's units go into FAILED, and gordon (being the only active pilot) is immediately getting new units assigned -- but those seem to go into FAILED immediately, too. at ~4700s, that pilot does, and the stampede pilot becomes available -- it gets the remaining units, and executes them nicely. Those are the ones which get staged out. Stampede gets a smaller share than gordon, because gordon already sucked up more CUs than 1/3rd.

So, scheduling wise thats ok. Trestles and Gordon are not... :(

[mturilli]

OK. Please feel free to change the title of this ticket accordingly.

I am afraid I will have to stop my tests on XSEDE. At the moment RP seems to be working only (and not always) with stampede.

[andre-merzky]

I am going to close this ticket then -- we have separate tickets for gordon (#309) and trestles (#301). I did not see a ticket on blacklight, and also have no login (or at least no access). If there are problems there, too, would you mind opening a separate ticket with some details, please? Also, I wasn't aware that stampede sometimes fails for you, and also could not find a specific ticket. Any details on whats up here?

Ole, Mark, I will need your input on those tickets I'm afraid. I think we all agree that there won't be a release before we have stable operation across XSEDE. I know its weekend, and Mark is offline anyways -- but lets please focus on those issues on Monday...

[mturilli]

I understand it is difficult to keep track of all the tickets/issues we discussed so here some clarifications that might be useful:

• We discussed on skype the problem with blacklight. The ticket is opened with saga-python (360) as the issue stems from a bug in the PBS adaptor, specifically in the parsing of pn values and the use of the ppm parameter. I count 5 comments from you on that ticket.
• The issues with stampede were reported by you to me :) I had some failures of which I informed you via skype and you told me that also Ole had had problems with stampede but it seemed a problem with the resource and not with RP. This is why I did not open a ticket about stampede.
• In full disclosure, I had once a problem with RP stalling - no logs produced, no activity triggered on the remote resources. I did not open a ticket because I would not know what to write. There were no logs, no jobs scheduled on the remote resources, a minimal sandbox with no logs. The resources involved were stampede, gordon, and trestles.

[andre-merzky]

Thanks for the clarification on blacklight and stampede. Now I see what you meant with stampede -- that seemed indeed unrelated to RP, fortunately, and by now stampede seems to be stable again.

Re the stalling run: if the minimal sandboxes existed (only containing the bootstrapped and pilot agent), then the transfer of those files via SAGA worked. Its hard to say what (did not) happen(ed) after that.

If RP hangs and you don't see any log activity, please interrupt via CTRL-C and post the complete resulting stacktrace and the application code (if we can run that) -- we may be able to track it down from there.

Thanks!