HashDist-installed nosetests script is broken on Cray XE6 (Garnet)

[ahmadia]

I get a "No profile found" error when trying to execute the nosetests script on Garnet. Here's a sample pbs script:

#!/bin/bash
#PBS -l select=16:ncpus=32:mpiprocs=32
#PBS -A ERDCV00898R40
#PBS -l walltime=001:00:00
#PBS -q debug
#PBS -N testProteus
#PBS -j oe
#PBS -l application=proteus

source /opt/modules/default/init/bash
module swap PrgEnv-pgi PrgEnv-gnu
module unload xt-libsci
module load acml
source /u/aron/proteus/garnet.gnu/bin/proteus_env.sh
export LD_LIBRARY_PATH=/u/aron/proteus/garnet.gnu/lib:${LD_LIBRARY_PATH}
PROTEUS_PYTHON=$(which python)
PROTEUS_NOSETESTS=$(which nosetests)

cd $PBS_O_WORKDIR
export MPICH_RANK_REORDER_METHOD=2
echo "testing gauges"
echo "++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++"
aprun -n 1 -N 1 ${PROTEUS_NOSETESTS} -v ../proteus/test/test_gauges.py

Here's the nosetests script itself:

#!/bin/sh
# -*- mode: python -*-
"true" '''\';i="python";if [ "$HDIST_IN_BUILD" = "yes" ] ; then exec "$i" "$0" "$@"; fi;o=`pwd`;p="$0";while true; do test -L "$p";il=$?;cd `dirname "$p"`;pdi
r=`pwd -P`;d="$pdir";while [ "$d" != / ]; do if [ -e "$d/artifact.json" ]; then if [ ! -e "$d/bin/$i" ]; then echo "Unable to locate needed $i in $p/bin";echo
 "HashDist profile $d has likely been corrupted, please try rebuilding.";exit 127;fi;cd "$o" && exec "$d/bin/$i" "$0" "$@";fi;cd ..;d=`pwd -P`;done;cd "$pdir"
;if [ "$il" -ne 0 ]; then echo "No profile found.";exit 127;fi;p=`readlink $p`;done;
''' # end multi-line shebang, see hashdist.core.build_tools
# EASY-INSTALL-ENTRY-SCRIPT: 'nose==1.3.4','console_scripts','nosetests'
__requires__ = 'nose==1.3.4'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('nose==1.3.4', 'console_scripts', 'nosetests')()
    )
# vi: filetype=python

[certik]

Can you post the whole stacktrace? I guess the error is coming from the echo "No profile found." in the script you posted? If so, we need to debug the shebang I assume. Any ideas what might be causing it?

[ahmadia]

I don't have time to get back to it this week. I've seen similar problems due to symbolic links and relative paths on the file system. That is the whole stack trace, the error is somewhere in the shebang script installation.

On Tuesday, January 6, 2015, Ondřej Čertík notifications@github.com wrote:

Can you post the whole stacktrace? I guess the error is coming from echo "No profile found." in the script you posted? If so, we need to debug the shebang I assume. Any ideas what might be causing it?

— Reply to this email directly or view it on GitHub https://github.com/hashdist/hashstack/issues/599#issuecomment-68921981.

[ahmadia]

By the way, the workaround is to do something like /path/to/python /path/to/nosetests, which works correctly since it doesn't require the shebang logic.