search

hammerlab / secotrec

Setup Coclobas/Ketrew Clusters
Apache License 2.0
5 stars 6 forks source link

Error when using many NFS servers #41

Closed tavinathanson closed 7 years ago

tavinathanson commented 7 years ago

I get this when I use more than approximately 20 mounted NFS servers:

tavi@tavi:~/coclo$ secotrec-gke status
WARNING: Command "kserver-compose" is too big for `sh -c <>`: 277920 B
secotrec-gke: internal error, uncaught exception:
              (Failure "Script too long")

I modified secotrec to output some of this script; each NFS server adds approximately the following gunk:

 { 'printf' 'SECOTREC: Mounted-tavi-cloud-nfs: Checking...\n' ; }  ; if {  {  { 'test' '-f' '/nfs-pool/.witness.txt' ; }  ; [ $? -eq 0 ] ; } ; } ; then  { 'printf' 'SECOTREC: Mounted-tavi-cloud-nfs: Already Done.\n' ; }  ; else  { 'printf' 'SECOTREC: Mounted-tavi-cloud-nfs: Build In Progress\n' ; }  ; if {  { { { { true &&  { (  eval "$(printf -- "exec %s>%s" 1 '/tmp/cmd-Mounting-tavi-cloud-nfs-stdout-0')" || { echo 'Exec "exec %s>%s" 1 '/tmp/cmd-Mounting-tavi-cloud-nfs-stdout-0' failed' >&2 ; }  ;  eval "$(printf -- "exec %s>%s" 2 '/tmp/cmd-Mounting-tavi-cloud-nfs-stderr-0')" || { echo 'Exec "exec %s>%s" 2 '/tmp/cmd-Mounting-tavi-cloud-nfs-stderr-0' failed' >&2 ; }  ;  { if {  { { { { true &&  { (  eval "$(printf -- "exec %s>%s" 1 '/tmp/cmd-apt-install-nfs-client-stdout-0')" || { echo 'Exec "exec %s>%s" 1 '/tmp/cmd-apt-install-nfs-client-stdout-0' failed' >&2 ; }  ;  eval "$(printf -- "exec %s>%s" 2 '/tmp/cmd-apt-install-nfs-client-stderr-0')" || { echo 'Exec "exec %s>%s" 2 '/tmp/cmd-apt-install-nfs-client-stderr-0' failed' >&2 ; }  ;  {  { 'sudo' 'apt-get' 'update' ; }   ; } ; ) ; [ $? -eq 0 ] ; } ; } &&  { (  eval "$(printf -- "exec %s>%s" 1 '/tmp/cmd-apt-install-nfs-client-stdout-1')" || { echo 'Exec "exec %s>%s" 1 '/tmp/cmd-apt-install-nfs-client-stdout-1' failed' >&2 ; }  ;  eval "$(printf -- "exec %s>%s" 2 '/tmp/cmd-apt-install-nfs-client-stderr-1')" || { echo 'Exec "exec %s>%s" 2 '/tmp/cmd-apt-install-nfs-client-stderr-1' failed' >&2 ; }  ;  {  { 'sudo' 'apt-get' 'upgrade' '--yes' ; }   ; } ; ) ; [ $? -eq 0 ] ; } ; } &&  { (  eval "$(printf -- "exec %s>%s" 1 '/tmp/cmd-apt-install-nfs-client-stdout-2')" || { echo 'Exec "exec %s>%s" 1 '/tmp/cmd-apt-install-nfs-client-stdout-2' failed' >&2 ; }  ;  eval "$(printf -- "exec %s>%s" 2 '/tmp/cmd-apt-install-nfs-client-stderr-2')" || { echo 'Exec "exec %s>%s" 2 '/tmp/cmd-apt-install-nfs-client-stderr-2' failed' >&2 ; }  ;  {  { 'sudo' 'apt-get' 'install' '--yes' 'nfs-client' ; }   ; } ; ) ; [ $? -eq 0 ] ; } ; } ; [ $? -eq 0 ] ; } ; } ; then : ; else  { 'printf' 'SECOTREC: apt-install-nfs-client; FAILED:\n\n' ; }  ;  { 'printf' '``````````stdout\n' ; }  ;  { 'cat' '/tmp/cmd-apt-install-nfs-client-stdout-0' ; }  ;  { 'printf' '\n``````````\n' ; }  ;  { 'printf' '``````````stderr\n' ; }  ;  { 'cat' '/tmp/cmd-apt-install-nfs-client-stderr-0' ; }  ;  { 'printf' '\n``````````\n' ; }  ;  { 'printf' '``````````stdout\n' ; }  ;  { 'cat' '/tmp/cmd-apt-install-nfs-client-stdout-1' ; }  ;  { 'printf' '\n``````````\n' ; }  ;  { 'printf' '``````````stderr\n' ; }  ;  { 'cat' '/tmp/cmd-apt-install-nfs-client-stderr-1' ; }  ;  { 'printf' '\n``````````\n' ; }  ;  { 'printf' '``````````stdout\n' ; }  ;  { 'cat' '/tmp/cmd-apt-install-nfs-client-stdout-2' ; }  ;  { 'printf' '\n``````````\n' ; }  ;  { 'printf' '``````````stderr\n' ; }  ;  { 'cat' '/tmp/cmd-apt-install-nfs-client-stderr-2' ; }  ;  { 'printf' '\n``````````\n' ; }  ;  { printf -- '%s\n' "EDSL.fail called" >&2 ; kill -s USR1 ${genspio_trap_19_33428} ; }  ; fi  ; } ; ) ; [ $? -eq 0 ] ; } ; } &&  { (  eval "$(printf -- "exec %s>%s" 1 '/tmp/cmd-Mounting-tavi-cloud-nfs-stdout-1')" || { echo 'Exec "exec %s>%s" 1 '/tmp/cmd-Mounting-tavi-cloud-nfs-stdout-1' failed' >&2 ; }  ;

@smondet says:

I'll find a way to pass that as a script instead of sh -c <cmd>

smondet commented 7 years ago

For the science; if I remove the check made by secotrec and let docker deal with the long cmd-line I get:

kserver_1    | panic: standard_init_linux.go:178: exec user process caused "argument list too long" [recovered]
kserver_1    |  panic: standard_init_linux.go:178: exec user process caused "argument list too long"
kserver_1    | 
kserver_1    | goroutine 1 [running, locked to thread]:
kserver_1    | panic(0x7e77c0, 0xc820320af0)
kserver_1    |  /usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
kserver_1    | github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction.func1(0xc8202c1300)
kserver_1    |  /build/runc-Fx_CW1/runc-1.0.0~rc2/_build/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:478 +0x38e
kserver_1    | panic(0x7e77c0, 0xc820320af0)
kserver_1    |  /usr/lib/go-1.6/src/runtime/panic.go:443 +0x4e9
kserver_1    | github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc8202c0c10, 0xc82002e088, 0xc8202c0d20)
kserver_1    |  /build/runc-Fx_CW1/runc-1.0.0~rc2/_build/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0x136
kserver_1    | github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc820078730, 0x7fa96f4b3a38, 0xc820320af0)
kserver_1    |  /build/runc-Fx_CW1/runc-1.0.0~rc2/_build/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x5b1
kserver_1    | main.glob.func8(0xc82009e780, 0x0, 0x0)
kserver_1    |  /build/runc-Fx_CW1/runc-1.0.0~rc2/_build/src/github.com/opencontainers/runc/main_unix.go:26 +0x68
kserver_1    | reflect.Value.call(0x74cd60, 0x8fddc8, 0x13, 0x8445f0, 0x4, 0xc8202c1280, 0x1, 0x1, 0x0, 0x0, ...)
kserver_1    |  /usr/lib/go-1.6/src/reflect/value.go:435 +0x120d
kserver_1    | reflect.Value.Call(0x74cd60, 0x8fddc8, 0x13, 0xc8202c1280, 0x1, 0x1, 0x0, 0x0, 0x0)
kserver_1    |  /usr/lib/go-1.6/src/reflect/value.go:303 +0xb1
kserver_1    | github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction(0x74cd60, 0x8fddc8, 0xc82009e780, 0x0, 0x0)
kserver_1    |  /build/runc-Fx_CW1/runc-1.0.0~rc2/_build/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:487 +0x2ee
kserver_1    | github.com/opencontainers/runc/vendor/github.com/urfave/cli.Command.Run(0x8474a8, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x8dd200, 0x51, 0x0, ...)
kserver_1    |  /build/runc-Fx_CW1/runc-1.0.0~rc2/_build/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/command.go:191 +0xfec
kserver_1    | github.com/opencontainers/runc/vendor/github.com/urfave/cli.(*App).Run(0xc820001980, 0xc82000a100, 0x2, 0x2, 0x0, 0x0)
kserver_1    |  /build/runc-Fx_CW1/runc-1.0.0~rc2/_build/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:240 +0xaa4
kserver_1    | main.main()
kserver_1    |  /build/runc-Fx_CW1/runc-1.0.0~rc2/_build/src/github.com/opencontainers/runc/main.go:132 +0xd98