shellspec / shellspec

A full-featured BDD unit testing framework for bash, ksh, zsh, dash and all POSIX shells
https://shellspec.info
MIT License
1.17k stars 73 forks source link

Segmentation fault (core dumped) with --kcov in circumstance on Travis linux xenial #214

Closed alexzhangs closed 3 years ago

alexzhangs commented 3 years ago

This issue is found only in the environment described below. It works totally well on macOS and AWS Linux 2 AMI (with the same shellspec and kcov version and test case).

Environment:

Operating System Details
  Distributor ID:   Ubuntu
  Description:  Ubuntu 16.04.6 LTS
  Release:  16.04
  Codename: xenial

/bin/bash [bash 4.3.48(1)-release]

$ shellspec --version
0.28.1

$ kcov --version
kcov 38

spec/foo_spec.sh:

Describe 'Foo'
  xsh () {
    function __xsh_clean () { unset -f $(echo __xsh_clean); :; }
    trap "trap - RETURN; __xsh_clean;" RETURN
    export XSH_HOME=~/.xsh
    export XSH_DEV_HOME=~/.xsh/lib-dev
  }
  export -f xsh
  exported_functions () { declare -Fx | awk '{print $3}'; }

  It 'call function: xsh'
    When call xsh
    The status should be success
    The output should include ''
    The variable XSH_HOME should be exported
    #The result of function exported_functions should include 'xsh'
  End
End

The Segmentation fault is caused by either of the last 2 statements: variable and result of function. My debug is focused on the variable statement. Make some slight change on xsh.sh, such as remove trap or __xsh_clean, the problem will go away.

Command:

~/.local/bin/shellspec --kcov --covdir ~/coverage -s /bin/bash spec/foo_spec.sh

Error log:

Running: /bin/bash [bash 4.3.48(1)-release]
/home/travis/.local/lib/shellspec/lib/core/dsl.sh: line 173:  5170 Segmentation fault      (core dumped) ( set -e; shift; case $# in 
    0)
        shellspec_invoke_example 3>&2
    ;;
    *)
        shellspec_invoke_example "$@" 3>&2
    ;;
esac )

Unexpected output to stderr occurred at line 5-10 in 'spec/foo_spec.sh'
F

Examples:
  1) Foo call function: xsh
     When call xsh

     1.1) Example aborted (exit status: 139)

          # spec/foo_spec.sh:5-11

Finished in 0.49 seconds (user 0.30 seconds, sys 0.05 seconds)
1 example, 1 failure

Failure examples / Errors: (Listed here affect your suite's status)

shellspec spec/foo_spec.sh:5 # 1) Foo call function: xsh FAILED

Code covered: 100.00%, Executed lines: 2, Instrumented lines: 2

Aborted with status code [executor: 0] [reporter: 101] [error handler: 102]
Fatal error occurred, terminated with exit status 102.
The command "~/.local/bin/shellspec --kcov --covdir ~/coverage -s /bin/bash spec/foo_spec.sh" exited with 102.

Done. Your build exited with 1.

Since Travis is the only environment the problem is found, the debug is very painful. I inject debug code into shellspec at the deploy time by .travis.yml.

sed -i '/shellspec_yield()/ r /dev/stdin' ~/.local/lib/shellspec/lib/core/dsl.sh <<< 'declare -f "shellspec_yield$SHELLSPEC_BLOCK_NO"'
sed -i 's/eval shellspec_syntax_dispatch modifier.*$/{ set -vx; &; declare ret=$?; set +vx; [[ $ret -eq 0 ]]; } 2>\&1/' ~/.local/lib/shellspec/lib/core/subjects/variable.sh
sed -i 's/shellspec_exists_envkey.*$/trap -p; echo $-; ulimit -a; set; { set -vx; &; declare ret=$?; set +vx; [[ $ret -eq 0 ]]; } 2>\&1/' ~/.local/lib/shellspec/lib/core/matchers/be/variable.sh

Step by step, finally found the code where the segmentation fault is triggered. Seems the error is triggered by the subshell within the function shellspec_exists_envkey.

shellspec_exists_envkey () 
{ 
    ( key="$1";
    function callback () 
    { 
        [ ! "$1" = "$key" ] && :
    };
    shellspec_list_envkeys callback && return 1;
    return 0 ) && :
}

I simplified the function code, and I can see the difference, but really don't understand why, don't have a clue what to do next, is the debugger like gdb is a choice?

# the problem goes away without subshell
shellspec_exists_envkey () { return 0; }

# the problem remains with subshell
shellspec_exists_envkey () { (return 0); }

The following is for reference.

The full .travis.yml:

language: bash

git:
  # make sure it's a full clone
  depth: false

addons:
  # for kcov
  apt:
    packages:
      - binutils-dev
      - libcurl4-openssl-dev
      - libdw-dev
      - libiberty-dev
  homebrew:
    update: true
    packages:
      - kcov

jobs:
  include:
    - os: linux
      dist: xenial
    - os: osx
      osx_image: xcode12.2
env:
  global:
    # shellspec and kcov
    - PATH=${HOME}/.local/bin:${HOME}/kcov/bin:${PATH}
    - COVDIR=${HOME}/coverage

before_install:
  # kcov: test coverage
  - |
    if [[ "$TRAVIS_OS_NAME" = "linux" ]]; then
      wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz
      tar xzf master.tar.gz
      cd kcov-master
      mkdir build
      cd build
      cmake -DCMAKE_INSTALL_PREFIX=${HOME}/kcov ..
      make
      make install
      cd ../..
      rm -rf kcov-master
    fi

  # shellspec: test framework
  - curl -fsSL https://git.io/shellspec | sh -s -- -y

    if [[ "$TRAVIS_OS_NAME" = "linux" ]]; then
    fi

install:
  # xsh
  - bash install.sh -s
  - source ~/.xshrc

script:
  - shellspec --version
  - kcov --version
  # start test with coverage
  - ~/.local/bin/shellspec --kcov --covdir ~/coverage -s /bin/bash spec/foo_spec.sh

after_success:
  # upload coverage report
  - bash <(curl -s https://codecov.io/bash) -s "${COVDIR}"

The system limits on Travis:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31806
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 30000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 31806
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

The full list of environment variables on Travis:

ANSI_CLEAR='\033[0K'
ANSI_GREEN='\033[32;1m'
ANSI_RED='\033[31;1m'
ANSI_RESET='\033[0m'
ANSI_YELLOW='\033[33;1m'
APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1
BASH=/bin/bash
BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_LINENO=([0]="24" [1]="73" [2]="89" [3]="55" [4]="73" [5]="56" [6]="20" [7]="22" [8]="56" [9]="73" [10]="56" [11]="17" [12]="56" [13]="73" [14]="73" [15]="67" [16]="40" [17]="355" [18]="285" [19]="70" [20]="57" [21]="241" [22]="207" [23]="52" [24]="147" [25]="49" [26]="75" [27]="57" [28]="139" [29]="28" [30]="81" [31]="0")
BASH_SOURCE=([0]="/home/travis/.local/lib/shellspec/lib/core/matchers/be/variable.sh" [1]="/home/travis/.local/lib/shellspec/lib/core/matchers.sh" [2]="/home/travis/.local/lib/shellspec/lib/general.sh" [3]="/home/travis/.local/lib/shellspec/lib/core/matchers/be/variable.sh" [4]="/home/travis/.local/lib/shellspec/lib/core/syntax.sh" [5]="/home/travis/.local/lib/shellspec/lib/general.sh" [6]="/home/travis/.local/lib/shellspec/lib/core/syntax.sh" [7]="/home/travis/.local/lib/shellspec/lib/core/matchers.sh" [8]="/home/travis/.local/lib/shellspec/lib/core/verb.sh" [9]="/home/travis/.local/lib/shellspec/lib/core/syntax.sh" [10]="/home/travis/.local/lib/shellspec/lib/general.sh" [11]="/home/travis/.local/lib/shellspec/lib/core/syntax.sh" [12]="/home/travis/.local/lib/shellspec/lib/core/subjects/variable.sh" [13]="/home/travis/.local/lib/shellspec/lib/core/syntax.sh" [14]="/home/travis/.local/lib/shellspec/lib/general.sh" [15]="/home/travis/.local/lib/shellspec/lib/general.sh" [16]="/home/travis/.local/lib/shellspec/lib/core/statement.sh" [17]="/home/travis/.local/lib/shellspec/lib/core/statement.sh" [18]="/home/travis/.local/lib/shellspec/lib/core/dsl.sh" [19]="/home/travis/.local/lib/shellspec/lib/core/dsl.sh" [20]="/tmp/shellspec.1619666467.5053/kcov/xsh [specfiles]" [21]="/home/travis/.local/lib/shellspec/lib/core/dsl.sh" [22]="/home/travis/.local/lib/shellspec/lib/core/dsl.sh" [23]="/home/travis/.local/lib/shellspec/lib/core/dsl.sh" [24]="/tmp/shellspec.1619666467.5053/kcov/xsh [specfiles]" [25]="/home/travis/.local/lib/shellspec/lib/core/dsl.sh" [26]="/tmp/shellspec.1619666467.5053/kcov/xsh [specfiles]" [27]="/tmp/shellspec.1619666467.5053/kcov/xsh [specfiles]" [28]="/home/travis/.local/lib/shellspec/lib/core/dsl.sh" [29]="/home/travis/.local/lib/shellspec/lib/core/dsl.sh" [30]="/tmp/shellspec.1619666467.5053/kcov/xsh [specfiles]" [31]="/tmp/shellspec.1619666467.5053/kcov/xsh [specfiles]")
BASH_VERSINFO=([0]="4" [1]="3" [2]="48" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
BASH_VERSION='4.3.48(1)-release'
CI=true
COMPOSER_NO_INTERACTION=1
CONTINUOUS_INTEGRATION=true
COVDIR=/home/travis/coverage
DEBIAN_FRONTEND=noninteractive
DIRSTACK=()
EUID=2000
FUNCNAME=([0]="shellspec_matcher__match" [1]="shellspec_matcher_do_match_positive" [2]="shellspec_matcher_do_match" [3]="shellspec_matcher_be_exported" [4]="shellspec_syntax_dispatch" [5]="shellspec_matcher_be" [6]="shellspec_syntax_dispatch" [7]="shellspec_matcher" [8]="shellspec_verb_should" [9]="shellspec_syntax_dispatch" [10]="shellspec_modifier_should" [11]="shellspec_syntax_dispatch" [12]="shellspec_subject_variable" [13]="shellspec_syntax_dispatch" [14]="shellspec_subject" [15]="shellspec_statement_subject" [16]="shellspec_statement_ordinal" [17]="shellspec_statement_preposition" [18]="shellspec_the" [19]="shellspec_statement" [20]="shellspec_yield2" [21]="shellspec_yield" [22]="shellspec_invoke_example" [23]="shellspec_example" [24]="shellspec_example2" [25]="shellspec_example_block" [26]="shellspec_block2" [27]="shellspec_yield1" [28]="shellspec_yield" [29]="shellspec_example_group" [30]="shellspec_block1" [31]="main")
GEM_HOME=/home/travis/.rvm/gems/ruby-2.5.3
GEM_PATH=/home/travis/.rvm/gems/ruby-2.5.3:/home/travis/.rvm/gems/ruby-2.5.3@global
GIT_ASKPASS=echo
GOPATH=/home/travis/gopath
GOROOT=/home/travis/.gimme/versions/go1.11.1.linux.amd64
GROUPS=()
HAS_ANTARES_THREE_LITTLE_FRONZIES_BADGE=true
HAS_JOSH_K_SEAL_OF_APPROVAL=true
HISTCONTROL=ignoredups:ignorespace
HISTFILESIZE=2000
HISTSIZE=1000
HOME=/home/travis
HOSTNAME=travis-job-4ef397bb-527f-4b71-b9b3-c29896b99e79
HOSTTYPE=x86_64
IFS=$' \t\n'
IRBRC=/home/travis/.rvm/rubies/ruby-2.5.3/.irbrc
JRUBY_OPTS=' --client -J-XX:+TieredCompilation -J-XX:TieredStopAtLevel=1 -J-Xss2m -Xcompile.invokedynamic=false'
KCOV_BASH_COMMAND=/bin/bash
KCOV_BASH_USE_DEBUG_TRAP=1
KCOV_BASH_XTRACEFD=7500
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_ALL=en_US.UTF-8
LC_CTYPE=en_US.UTF-8
LINENO=24
LOGNAME=travis
MACHTYPE=x86_64-pc-linux-gnu
MAIL=/var/mail/travis
MANPATH=/home/travis/.rvm/rubies/ruby-2.5.3/share/man:/usr/local/cmake-3.12.4/man:/usr/local/clang-7.0.0/share/man:/usr/local/man:/usr/local/share/man:/usr/share/man:/home/travis/.rvm/man
MERB_ENV=test
MY_RUBY_HOME=/home/travis/.rvm/rubies/ruby-2.5.3
NVM_CD_FLAGS=
NVM_DIR=/home/travis/.nvm
NVM_RC_VERSION=
OLDPWD=/home/travis/build/alexzhangs/xsh
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PAGER=cat
PATH=/tmp/shellspec.1619666467.5053/1:/home/travis/build/alexzhangs/xsh/spec/support/bin:/home/travis/.local/bin:/home/travis/kcov/bin:/home/travis/bin:/home/travis/.local/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin
PERLBREW_HOME=/home/travis/.perlbrew
PERLBREW_ROOT=/home/travis/perl5/perlbrew
PERLBREW_SHELLRC_VERSION=0.86
PIPESTATUS=([0]="0")
PPID=5156
PS4=+
PWD=/home/travis/build/alexzhangs/xsh
PYENV_ROOT=/opt/pyenv
PYENV_SHELL=bash
PYTHON_CFLAGS='-g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security'
PYTHON_CONFIGURE_OPTS='--enable-unicode=ucs4 --with-wide-unicode --enable-shared --enable-ipv6 --enable-loadable-sqlite-extensions --with-computed-gotos'
RACK_ENV=test
RAILS_ENV=test
RBENV_SHELL=bash
RUBY_VERSION=ruby-2.5.3
SHELL=/bin/bash
SHELLOPTS=braceexpand:errexit:functrace:hashall:interactive-comments
SHELLSPEC_ACK=$'\006'
SHELLSPEC_AFTER_ALL_INDEX=0
SHELLSPEC_AFTER_CALL_INDEX=0
SHELLSPEC_AFTER_EACH_INDEX=0
SHELLSPEC_AFTER_MOCK_INDEX=0
SHELLSPEC_AFTER_RUN_INDEX=0
SHELLSPEC_AUX_LINENO=
SHELLSPEC_BANNER=1
SHELLSPEC_BANNER_FILE=/home/travis/build/alexzhangs/xsh/spec/banner
SHELLSPEC_BEFORE_ALL_INDEX=0
SHELLSPEC_BEFORE_CALL_INDEX=0
SHELLSPEC_BEFORE_EACH_INDEX=0
SHELLSPEC_BEFORE_MOCK_INDEX=0
SHELLSPEC_BEFORE_RUN_INDEX=0
SHELLSPEC_BEL=$'\a'
SHELLSPEC_BLOCK_NO=2
SHELLSPEC_BS=$'\b'
SHELLSPEC_BUILTIN_PRINT=
SHELLSPEC_BUILTIN_PRINTF=1
SHELLSPEC_BUILTIN_TYPESETF=1
SHELLSPEC_BUSYBOX_W32=
SHELLSPEC_CAN=$'\030'
SHELLSPEC_CHMOD=/bin/chmod
SHELLSPEC_CLONE_TYPE=bash
SHELLSPEC_COLOR=1
SHELLSPEC_COMPOUNDS=:shellspec_subject_entire:shellspec_modifier_entire:shellspec_matcher_be_empty:shellspec_matcher_be_block:shellspec_matcher_be_character:shellspec_matcher_end:shellspec_matcher_start:
SHELLSPEC_COUNT_FILE=
SHELLSPEC_COVERAGEDIR=/home/travis/coverage
SHELLSPEC_COVERAGE_KSH_WORKAROUND=
SHELLSPEC_COVERAGE_SETUP=/home/travis/.local/lib/shellspec/lib/cov/kcov.sh
SHELLSPEC_COVERAGE_SHELL_OPTIONS=
SHELLSPEC_CR=$'\r'
SHELLSPEC_CWD=/home/travis/build/alexzhangs/xsh
SHELLSPEC_DATA=
SHELLSPEC_DATE=/bin/date
SHELLSPEC_DC1=$'\021'
SHELLSPEC_DC2=$'\022'
SHELLSPEC_DC3=$'\023'
SHELLSPEC_DC4=$'\024'
SHELLSPEC_DEBUG_TRAP=1
SHELLSPEC_DEFAULT_PATH=spec
SHELLSPEC_DEFECT_BUILTIN=
SHELLSPEC_DEFECT_EMPTYPARAMS=
SHELLSPEC_DEFECT_ERREXIT=
SHELLSPEC_DEFECT_EXPORTP=
SHELLSPEC_DEFECT_READONLY=
SHELLSPEC_DEFECT_REDEFINE=
SHELLSPEC_DEFECT_SANDBOX=
SHELLSPEC_DEFECT_SETE=
SHELLSPEC_DEFECT_SHELLFLAG=
SHELLSPEC_DEFECT_SIGNAL=
SHELLSPEC_DEFECT_SUBSHELL=1
SHELLSPEC_DEFECT_XTRACE=
SHELLSPEC_DEFECT_ZSHEXIT=
SHELLSPEC_DEPRECATION_LOG=1
SHELLSPEC_DEPRECATION_LOGFILE=/tmp/shellspec.1619666467.5053/.shellspec-deprecation.log
SHELLSPEC_DEREFERENCE=
SHELLSPEC_DESCRIPTION=$'Foo\vcall function: xsh'
SHELLSPEC_DEV_TTY=/dev/tty
SHELLSPEC_DLE=$'\020'
SHELLSPEC_DOCKER_IMAGE=
SHELLSPEC_DRYRUN=
SHELLSPEC_EM=$'\031'
SHELLSPEC_ENABLED=1
SHELLSPEC_ENQ=$'\005'
SHELLSPEC_ENV=/usr/bin/env
SHELLSPEC_ENV_FILE=/home/travis/.local/lib/shellspec/lib/cov/kcov/.bashenv
SHELLSPEC_ENV_FROM=
SHELLSPEC_EOT=$'\004'
SHELLSPEC_ERREXIT=+e
SHELLSPEC_ERROR_EXIT_CODE=102
SHELLSPEC_ERROR_FILE=/tmp/shellspec.1619666467.5053/1/error
SHELLSPEC_ESC=$'\E'
SHELLSPEC_ETB=$'\027'
SHELLSPEC_ETX=$'\003'
SHELLSPEC_EVAL=$'shellspec_matcher__failure_message_when_negated() {\nshellspec_putsn "The specified variable is exported"'
SHELLSPEC_EVALUATION='When call xsh'
SHELLSPEC_EXAMPLE_COUNT=
SHELLSPEC_EXAMPLE_FILTER=
SHELLSPEC_EXAMPLE_ID=1-1
SHELLSPEC_EXAMPLE_NO=1
SHELLSPEC_EXECDIR=@project
SHELLSPEC_EXPECTATION='The variable XSH_HOME should be exported'
SHELLSPEC_FAILGLOB_AVAILABLE=1
SHELLSPEC_FAILURE_EXIT_CODE=101
SHELLSPEC_FAIL_FAST_COUNT=
SHELLSPEC_FAIL_LOW_COVERAGE=
SHELLSPEC_FAIL_NO_EXAMPLES=
SHELLSPEC_FF=$'\f'
SHELLSPEC_FILTER=1
SHELLSPEC_FIND=/usr/bin/find
SHELLSPEC_FOCUSED=
SHELLSPEC_FOCUS_FILTER=
SHELLSPEC_FORMATTER=progress
SHELLSPEC_FS=$'\034'
SHELLSPEC_GENERATORS=
SHELLSPEC_GRAMMAR_BLOCKS=
SHELLSPEC_GRAMMAR_DIRECTIVES=
SHELLSPEC_GRAMMAR_DSLS=
SHELLSPEC_GROUP_ID=1
SHELLSPEC_GS=$'\035'
SHELLSPEC_HELPERDIR=/home/travis/build/alexzhangs/xsh/spec
SHELLSPEC_HOOK=
SHELLSPEC_HOOK_ERROR=
SHELLSPEC_HOSTNAME=localhost
SHELLSPEC_HT=$'\t'
SHELLSPEC_IFS=$' \t\n'
SHELLSPEC_INFILE=file
SHELLSPEC_INFO=
SHELLSPEC_INSPECTION=/home/travis/.local/lib/shellspec/libexec/shellspec-inspection.sh
SHELLSPEC_INTERCEPTOR='|'
SHELLSPEC_KCOV=1
SHELLSPEC_KCOV_COMMON_OPTS='--include-path=. --include-pattern=.sh --exclude-pattern=/.shellspec,/spec/,/coverage/,/report/ --path-strip-level=1 '
SHELLSPEC_KCOV_COMPATIBLE_SHELL=1
SHELLSPEC_KCOV_FILENAME='xsh [specfiles]'
SHELLSPEC_KCOV_IN_FILE='/tmp/shellspec.1619666467.5053/kcov/xsh [specfiles]'
SHELLSPEC_KCOV_OPTS=--include-pattern=xsh.sh
SHELLSPEC_KCOV_PATH=kcov
SHELLSPEC_KCOV_VERSION='kcov 38'
SHELLSPEC_KEEP_TMPDIR=
SHELLSPEC_LEAK_FILE=/tmp/shellspec.1619666467.5053/1/1-1.leak
SHELLSPEC_LF=$'\n'
SHELLSPEC_LIB=/home/travis/.local/lib/shellspec/lib
SHELLSPEC_LIBEXEC=/home/travis/.local/lib/shellspec/libexec
SHELLSPEC_LINENO=9
SHELLSPEC_LINENO_BEGIN=5
SHELLSPEC_LINENO_END=11
SHELLSPEC_LIST=
SHELLSPEC_LOAD_PATH=/home/travis/build/alexzhangs/xsh/spec:/home/travis/.local/lib/shellspec/lib:/home/travis/.local/lib/shellspec/lib/libexec/reporter
SHELLSPEC_LOGFILE=/dev/tty
SHELLSPEC_LS=/bin/ls
SHELLSPEC_MARK_0=1
SHELLSPEC_MARK_1=1
SHELLSPEC_META=variable:XSH_HOME
SHELLSPEC_MOCK_BINDIR=/tmp/shellspec.1619666467.5053/1
SHELLSPEC_MODE=runner
SHELLSPEC_MSLEEP=1
SHELLSPEC_MV=/bin/mv
SHELLSPEC_NAK=$'\025'
SHELLSPEC_NOEXEC_TMPDIR=
SHELLSPEC_NOMATCH_AVAILABLE=
SHELLSPEC_OPTIONS=
SHELLSPEC_ORDINAL_eighteenth=18
SHELLSPEC_ORDINAL_eighth=8
SHELLSPEC_ORDINAL_eleventh=11
SHELLSPEC_ORDINAL_fifteenth=15
SHELLSPEC_ORDINAL_fifth=5
SHELLSPEC_ORDINAL_first=1
SHELLSPEC_ORDINAL_fourteenth=14
SHELLSPEC_ORDINAL_fourth=4
SHELLSPEC_ORDINAL_nineteenth=19
SHELLSPEC_ORDINAL_ninth=9
SHELLSPEC_ORDINAL_second=2
SHELLSPEC_ORDINAL_seventeenth=17
SHELLSPEC_ORDINAL_seventh=7
SHELLSPEC_ORDINAL_sixteenth=16
SHELLSPEC_ORDINAL_sixth=6
SHELLSPEC_ORDINAL_tenth=10
SHELLSPEC_ORDINAL_third=3
SHELLSPEC_ORDINAL_thirteenth=13
SHELLSPEC_ORDINAL_twelfth=12
SHELLSPEC_ORDINAL_twentieth=20
SHELLSPEC_ORDINAL_zeroth=0
SHELLSPEC_OUTPUT_FD=9
SHELLSPEC_PATH=/home/travis/.local/bin:/home/travis/kcov/bin:/home/travis/bin:/home/travis/.local/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin
SHELLSPEC_PATHSEP=:
SHELLSPEC_PATH_ALIAS='|'
SHELLSPEC_PATH_IS_READONLY=
SHELLSPEC_PATTERN='*_spec.sh'
SHELLSPEC_PENDING_MESSAGE=verbose
SHELLSPEC_PENDING_REASON=
SHELLSPEC_PRECHECKER_STATUS=/tmp/shellspec.1619666467.5053/.shellspec-prechecker.status
SHELLSPEC_PRINTF=printf
SHELLSPEC_PROC_VERSION=/proc/version
SHELLSPEC_PROFILER=
SHELLSPEC_PROFILER_LIMIT=10
SHELLSPEC_PROFILER_LOG=/tmp/shellspec.1619666467.5053/.shellspec-profiler.log
SHELLSPEC_PROFILER_REPORT=/home/travis/build/alexzhangs/xsh/report/profiler.log
SHELLSPEC_PROFILER_SIGNAL=/tmp/shellspec.1619666467.5053/.shellspec-profiler.signal
SHELLSPEC_PROJECT_NAME=xsh
SHELLSPEC_PROJECT_ROOT=/home/travis/build/alexzhangs/xsh
SHELLSPEC_QUICK=
SHELLSPEC_QUICK_FILE=/home/travis/build/alexzhangs/xsh/.shellspec-quick.log
SHELLSPEC_RANDOM=
SHELLSPEC_REPAIR=
SHELLSPEC_REPORTDIR=/home/travis/build/alexzhangs/xsh/report
SHELLSPEC_REPORTERLIB=/home/travis/.local/lib/shellspec/lib/libexec/reporter
SHELLSPEC_REPORTER_PID=/tmp/shellspec.1619666467.5053/.shellspec-reporter.pid
SHELLSPEC_REQUIRES=spec_helper
SHELLSPEC_RM=/bin/rm
SHELLSPEC_ROOT=/home/travis/.local/lib/shellspec
SHELLSPEC_RS=$'\036'
SHELLSPEC_SANDBOX=
SHELLSPEC_SANDBOX_PATH=
SHELLSPEC_SEED=
SHELLSPEC_SELF=/home/travis/.local/lib/shellspec/shellspec
SHELLSPEC_SET_OPTION=
SHELLSPEC_SHEBANG_MULTIARG=
SHELLSPEC_SHELL=/bin/bash
SHELLSPEC_SHELL_OPTIONS=
SHELLSPEC_SHELL_TYPE=bash
SHELLSPEC_SHELL_VERSION='4.3.48(1)-release'
SHELLSPEC_SHOPT_AVAILABLE=1
SHELLSPEC_SH_VERSION=
SHELLSPEC_SI=$'\017'
SHELLSPEC_SKIP_ID=
SHELLSPEC_SKIP_MESSAGE=verbose
SHELLSPEC_SKIP_REASON=
SHELLSPEC_SLEEP=/bin/sleep
SHELLSPEC_SO=$'\016'
SHELLSPEC_SOH=$'\001'
SHELLSPEC_SORT=/usr/bin/sort
SHELLSPEC_SOURCE=/home/travis/.local/lib/shellspec/lib/core/matchers/satisfy.sh
SHELLSPEC_SPECDIR=/home/travis/build/alexzhangs/xsh/spec
SHELLSPEC_SPECFILE=spec/foo_spec.sh
SHELLSPEC_SPEC_NO=1
SHELLSPEC_STATUS=0
SHELLSPEC_STDERR=
SHELLSPEC_STDERR_FILE=/tmp/shellspec.1619666467.5053/1/1-1.stderr
SHELLSPEC_STDIN_DEV=/dev/tty
SHELLSPEC_STDIN_FILE=/tmp/shellspec.1619666467.5053/1/1-1.stdin
SHELLSPEC_STDIO_FILE_BASE=/tmp/shellspec.1619666467.5053/1/1-1
SHELLSPEC_STDOUT=
SHELLSPEC_STDOUT_FILE=/tmp/shellspec.1619666467.5053/1/1-1.stdout
SHELLSPEC_STX=$'\002'
SHELLSPEC_SUB=$'\032'
SHELLSPEC_SUBJECT=/home/travis/.xsh
SHELLSPEC_SUPPORT_BIN=/home/travis/.local/lib/shellspec/lib/support-bin.sh
SHELLSPEC_SUPPORT_BINDIR=/home/travis/build/alexzhangs/xsh/spec/support/bin
SHELLSPEC_SW_EVALUATION=1
SHELLSPEC_SW_EXPECTATION=1
SHELLSPEC_SW_FAILED=
SHELLSPEC_SW_LEAK=
SHELLSPEC_SW_MATCHED=
SHELLSPEC_SW_NOT_IMPLEMENTED=
SHELLSPEC_SW_SYNTAX_ERROR=
SHELLSPEC_SW_UNHANDLED_STATUS=
SHELLSPEC_SW_UNHANDLED_STDERR=
SHELLSPEC_SW_UNHANDLED_STDOUT=
SHELLSPEC_SW_WARNED=
SHELLSPEC_SYN=$'\026'
SHELLSPEC_SYNTAXES=:shellspec_evaluation_call:shellspec_evaluation_run:shellspec_subject_entire:shellspec_subject_line:shellspec_subject_path:shellspec_subject_file:shellspec_subject_dir:shellspec_subject_directory:shellspec_subject_status:shellspec_subject_stderr:shellspec_subject_error:shellspec_subject_entire_stderr:shellspec_subject_entire_error:shellspec_subject_stdout:shellspec_subject_output:shellspec_subject_entire_stdout:shellspec_subject_entire_output:shellspec_subject_value:shellspec_subject_function:shellspec_subject_variable:shellspec_subject_word:shellspec_modifier_should:shellspec_modifier_entire:shellspec_modifier_contents:shellspec_modifier_entire_contents:shellspec_modifier_length:shellspec_modifier_line:shellspec_modifier_lines:shellspec_modifier_result:shellspec_modifier_word:shellspec_verb_should:shellspec_verb_should_not:shellspec_matcher_be:shellspec_matcher_be_empty_file:shellspec_matcher_be_empty_directory:shellspec_matcher_be_empty_dir:shellspec_matcher_be_empty:shellspec_matcher_be_exist:shellspec_matcher_be_file:shellspec_matcher_be_directory:shellspec_matcher_be_symlink:shellspec_matcher_be_pipe:shellspec_matcher_be_socket:shellspec_matcher_be_readable:shellspec_matcher_be_writable:shellspec_matcher_be_executable:shellspec_matcher_be_block:shellspec_matcher_be_block_device:shellspec_matcher_be_character:shellspec_matcher_be_character_device:shellspec_matcher_be_success:shellspec_matcher_be_failure:shellspec_matcher_be_valid_number:shellspec_matcher_be_valid_funcname:shellspec_matcher_be_valid:shellspec_matcher_be_defined:shellspec_matcher_be_undefined:shellspec_matcher_be_blank:shellspec_matcher_be_present:shellspec_matcher_be_exported:shellspec_matcher_be_readonly:shellspec_matcher_be_successful:shellspec_matcher_end_with:shellspec_matcher_end:shellspec_matcher_equal:shellspec_matcher_eq:shellspec_matcher_has:shellspec_matcher_has_setgid:shellspec_matcher_has_setuid:shellspec_matcher_include:shellspec_matcher_match:shellspec_matcher_match_pattern:shellspec_matcher_start_with:shellspec_matcher_start:shellspec_matcher_satisfy:
SHELLSPEC_SYNTAX_NAME=shellspec_matcher_be_exported
SHELLSPEC_TAB=$'\t'
SHELLSPEC_TAG_FILTER=
SHELLSPEC_TIME='time -p'
SHELLSPEC_TIME_LOG=/tmp/shellspec.1619666467.5053/.shellspec-time.log
SHELLSPEC_TMPBASE=/tmp/shellspec.1619666467.5053
SHELLSPEC_TMPDIR=/tmp
SHELLSPEC_TRAP=trap
SHELLSPEC_TTY=1
SHELLSPEC_UNIXTIME=1619666467
SHELLSPEC_UNREADONLY_PATH=/home/travis/.local/lib/shellspec/libexec/shellspec-unreadonly-path.sh
SHELLSPEC_US=$'\037'
SHELLSPEC_VARS_FILE=/tmp/shellspec.1619666467.5053/1/1-1.vars
SHELLSPEC_VERSION=0.28.1
SHELLSPEC_VT=$'\v'
SHELLSPEC_WARNING_AS_FAILURE=1
SHELLSPEC_WORKDIR=/tmp/shellspec.1619666467.5053/1
SHELLSPEC_WORKERS=0
SHELLSPEC_XTRACE=
SHELLSPEC_XTRACEFD=2
SHELLSPEC_XTRACEFD_VAR=BASH_XTRACEFD
SHELLSPEC_XTRACE_FILE=/tmp/shellspec.1619666467.5053/1/1-1.trace
SHELLSPEC_XTRACE_OFF=': @SHELLSPEC_XTRACE_OFF@; set +x'
SHELLSPEC_XTRACE_ON='BASH_XTRACEFD=$SHELLSPEC_XTRACEFD; set -x'
SHELLSPEC_XTRACE_ONLY=
SHLVL=5
SSH_CLIENT='10.16.7.9 43724 22'
SSH_CONNECTION='10.16.7.9 43724 10.30.0.50 22'
SSH_TTY=/dev/pts/0
TERM=xterm
TRAVIS=true
TRAVIS_ALLOW_FAILURE=
TRAVIS_APP_HOST=build.travis-ci.com
TRAVIS_APT_PROXY=http://build-cache.travisci.net
TRAVIS_ARCH=amd64
TRAVIS_BRANCH=kcov
TRAVIS_BUILD_DIR=/home/travis/build/alexzhangs/xsh
TRAVIS_BUILD_ID=224456940
TRAVIS_BUILD_NUMBER=175
TRAVIS_BUILD_STAGE_NAME=
TRAVIS_BUILD_WEB_URL=https://travis-ci.com/alexzhangs/xsh/builds/224456940
TRAVIS_CMD='~/.local/bin/shellspec --kcov --covdir ~/coverage -s /bin/bash spec/foo_spec.sh'
TRAVIS_COMMIT=bf155c2ebcebc1b375c03634a232e1ffd676e51d
TRAVIS_COMMIT_MESSAGE='Update .travis.yml'
TRAVIS_COMMIT_RANGE=08b67f5bc800...bf155c2ebceb
TRAVIS_CPU_ARCH=amd64
TRAVIS_DIST=xenial
TRAVIS_ENABLE_INFRA_DETECTION=true
TRAVIS_EVENT_TYPE=push
TRAVIS_HOME=/home/travis
TRAVIS_INFRA=unknown
TRAVIS_INIT=systemd
TRAVIS_INTERNAL_RUBY_REGEX='^ruby-(2\.[0-4]\.[0-9]|1\.9\.3)'
TRAVIS_JOB_ID=501868046
TRAVIS_JOB_NAME=
TRAVIS_JOB_NUMBER=175.1
TRAVIS_JOB_WEB_URL=https://travis-ci.com/alexzhangs/xsh/jobs/501868046
TRAVIS_LANGUAGE=shell
TRAVIS_OSX_IMAGE=
TRAVIS_OS_NAME=linux
TRAVIS_PRE_CHEF_BOOTSTRAP_TIME=2019-03-25T16:16:41
TRAVIS_PULL_REQUEST=false
TRAVIS_PULL_REQUEST_BRANCH=
TRAVIS_PULL_REQUEST_SHA=
TRAVIS_PULL_REQUEST_SLUG=
TRAVIS_REPO_SLUG=alexzhangs/xsh
TRAVIS_ROOT=/
TRAVIS_SECURE_ENV_VARS=false
TRAVIS_STACK_FEATURES='basic disabled-ipv6 docker docker-compose go-toolchain perl_interpreter perlbrew python_interpreter ruby_interpreter'
TRAVIS_STACK_JOB_BOARD_REGISTER=/.job-board-register.yml
TRAVIS_STACK_LANGUAGES='__stevonnie__ bash minimal sh shell'
TRAVIS_STACK_NAME=stevonnie
TRAVIS_STACK_NODE_ATTRIBUTES=/.node-attributes.yml
TRAVIS_STACK_TIMESTAMP='2019-03-25 16:16:53 UTC'
TRAVIS_SUDO=true
TRAVIS_TAG=
TRAVIS_TEST_RESULT=0
TRAVIS_TIMER_ID=09a87efe
TRAVIS_TIMER_START_TIME=1619666467673637296
TRAVIS_TMPDIR=/tmp/tmp.WgBpAmrAg7
TRAVIS_UID=2000
TZ=UTC
UID=2000
USER=travis
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
XDG_RUNTIME_DIR=/run/user/2000
XDG_SESSION_ID=2
XSH_DEV_HOME=/home/travis/.xsh/lib-dev
XSH_HOME=/home/travis/.xsh
_=ehBT
_system_arch=x86_64
_system_name=Ubuntu
_system_type=Linux
_system_version=16.04
rvm_bin_path=/home/travis/.rvm/bin
rvm_path=/home/travis/.rvm
rvm_prefix=/home/travis
rvm_version='1.29.7 (latest)'
shellspec_coverage=0
shellspec_i=6
shellspec_j=1
shellspec_k=1
shellspec_output_buf=$'\n'
shellspec_v=should
shellspec_work=' "${1}"  "${2}"  "${3}" "${4}" "${5}"'
ko1nksm commented 3 years ago

Thank you for investigating.

The CI environment is different from the actual machine, so it may behave strangely in rare cases. I've had a few such experiences. As you know, it is very difficult to resolve it. And travis is no longer supporting the OSS project, so I can't test. I would be applicate if you could help me.

is the debugger like gdb is a choice?

Unfortunately, I don't know how to test using gdb.

shellspec_exists_envkey() {
  (
    key="$1"
    callback() { [ ! "$1" = "$key" ] &&:; }
    shellspec_list_envkeys callback && return 1
    return 0
  ) &&:
}

This code is very strange and may be resolved by changing it to a more natural code.

I am wondering if subshells are really the cause of the problem. What happens if we don't use return?

# the problem goes away without subshell
shellspec_exists_envkey () { return 0; }

# the problem remains with subshell
shellspec_exists_envkey () { (return 0); }

# ?
shellspec_exists_envkey () { ( : ); }

# ?
shellspec_exists_envkey () { ( exit 0 ); }

If all the code using subshells fails, then implement it without subshells.

shellspec_exists_envkey() {
  shellspec_exists_envkey_key=$1
  shellspec_list_envkeys shellspec_exists_envkey_ && return 1
  return 0
}
shellspec_exists_envkey_() { [ ! "$1" = "$shellspec_exists_envkey_key" ]; }

I believe this code will work. However, I have not tested it with other shells, so additional modifications may be needed.

alexzhangs commented 3 years ago

And travis is no longer supporting the OSS project, so I can't test. I would be applicate if you could help me.

I'm able to get free credit for OSS projects by sending a request email to Travis suppot team. https://docs.travis-ci.com/user/billing-faq#what-if-i-am-building-open-source

And yes, I'm considering to request a SSH permission for the testing project to debug this issue.

I am wondering if subshells are really the cause of the problem. What happens if we don't use return?

I'll try your sample testing code. But I think the segementation fault is happening right on the subshell is being forked. because there's nothing printed in the trace after that. And I found another segmentation fault test case with a single status statement. I didn't look into it yet, but I knew that subshell is used in the code which were being tested.

I'll let you know if have made any progress.

alexzhangs commented 3 years ago

@ko1nksm I have got a local debug environment to reproduce this issue by running the Travis docker image locally.

Here are the steps:

Assuming you already have Docker app installed at your local (Mine is Docker Desktop 3.2.1 on macOS 11.2.1).

  1. Start a 'Travis xenial Ubuntu Linux` docker container at local:

    BUILDID="build-$RANDOM"
    INSTANCE="travisci/ci-stevonnie:packer-1564744294-e0797511"
    docker run --name $BUILDID -dit $INSTANCE /sbin/init
    docker exec -it $BUILDID bash -l
  2. Reproduce the issue inside the docker.

    
    su - travis
    sudo apt-get update -y

install the dependencies of kcov

sudo -E apt-get -yq --no-install-suggests --no-install-recommends install binutils-dev libcurl4-openssl-dev libdw-dev libiberty-dev

export PATH=${HOME}/.local/bin:${HOME}/kcov/bin:${PATH}

install shellspec

curl -fsSL https://git.io/shellspec | sh -s -- -y

install kcov

wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz tar xzf master.tar.gz (cd kcov-master && mkdir -p build && cd build && cmake -DCMAKE_INSTALL_PREFIX=${HOME}/kcov ..; make && make install)

shellspec --version kcov --version

tell the system to dump core file

ulimit -c unlimited

get shellspec testcase

git clone https://github.com/alexzhangs/xsh.git alexzhangs/xsh cd alexzhangs/xsh

trigger the segmentation fault

shellspec --kcov -s /bin/bash spec/foo_spec.sh


3. debug the dumped core file with gdb inside the docker

install gdb

sudo apt-get install gdb

install debug symbols for bash

U=http://ddebs.ubuntu.com D=$(lsb_release -cs) cat <<EOF | sudo tee /etc/apt/sources.list.d/ddebs.list deb ${U} ${D} main restricted universe multiverse

deb ${U} ${D}-security main restricted universe multiverse

deb ${U} ${D}-updates main restricted universe multiverse deb ${U} ${D}-proposed main restricted universe multiverse EOF wget -O - http://ddebs.ubuntu.com/dbgsym-release-key.asc | sudo apt-key add - sudo apt-get update -y sudo apt-get install -y bash-dbgsym

start to debug the dumped core file

gdb bash ./core


The gdb output:

travis@75e894a0623b:~/alexzhangs/xsh$ gdb bash core GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1 Copyright (C) 2016 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from bash...Reading symbols from /usr/lib/debug//bin/bash...done. done. [New LWP 7784] Core was generated by `/bin/bash /tmp/shellspec.1620732472.7667/kcov/xsh [specfiles]'. Program terminated with signal SIGSEGV, Segmentation fault.

0 discard_pipeline (chain=chain@entry=0x25a3888) at .././jobs.c:1090

1090 .././jobs.c: No such file or directory.



Please ignore the message `.././jobs.c: No such file or directory.`, I didn't install the bash source at my local.

The `bt` command output of `gdb` can be found at: https://justpaste.it/5bfo7
The `bt full` command output of `gdb` can be found at https://justpaste.it/4bijv

I'm not familier with gdb, the further investagition and help are needed.
alexzhangs commented 3 years ago

@ko1nksm Plus my last comment, I also tested the 3 piece of code you suggested, they all go to the Segmentation Fault sooner or later once meet a subshell.

?

shellspec_exists_envkey () { ( : ); }

?

shellspec_exists_envkey () { ( exit 0 ); }

shellspec_exists_envkey() { shellspec_exists_envkey_key=$1 shellspec_list_envkeys shellspec_existsenvkey && return 1 return 0 } shellspec_existsenvkey() { [ ! "$1" = "$shellspec_exists_envkey_key" ]; }

ko1nksm commented 3 years ago

@alexzhangs Thanks for writing how to run the travis Docker image locally. This helped me a lot!

Upon investigation, I have confirmed that this is a bug in bash and not a problem with travis, kcov, or shellspec. This bug seems to have been introduced in bash 4.3.2 and fixed in 4.4.0.

Here are the steps to reproduce it.

$ docker run -it fidian/multishell

root@469b51e73506:/# cat <<'HERE' > issue.sh
#!/bin/bash

xsh () {
  xsh_clean() { : $(echo); :; }
  trap "trap - RETURN; xsh_clean" RETURN
}

(
  # Required to enable coverage (kcov).
  set -o functrace
  trap ':' DEBUG

  xsh
  ( : ) # Segmentation fault
  echo end
)
HERE

root@469b51e73506:/# bash-4.3.2 issue.sh
issue.sh: line 16:    15 Segmentation fault      ( set -o functrace; trap ':' DEBUG; xsh; ( : ); echo end )

Currently, there is no way to work around this bug on the ShellSpec side. I suspect it will be difficult.

BTW, I figured out this bug by reducing the ShellSpec code from a reproducible environment.

alexzhangs commented 3 years ago

@ko1nksm Thanks for the amazing investigation! I was lost in the gdb debugging maze. ;)

I have moved to Travis dist bionic which has bash 4.4.20 packed inside. And all test cases go well.

Thanks again for the help! I'm closing this issue.