search

concourse / concourse-bosh-deployment

A toolchain for deploying Concourse with BOSH.
Apache License 2.0
84 stars 155 forks source link

Web instance is not running after update. (pthread_create failed) #252

Closed eyett closed 1 year ago

eyett commented 1 year ago

Describe the bug

Trying to deploy the latest concourse deployment. Web instance is failing with the below error

Task 3042 | 11:35:34 | Updating instance web: web/d13ff23a-e367-482c-952d-cbee90e21157 (0) `(canary)` (00:01:15)
                     L Error: 'web/d13ff23a-e367-482c-952d-cbee90e21157 (0)' is not running after update. Review logs for failed jobs: web
Task 3042 | 11:35:35 | Error: 'web/d13ff23a-e367-482c-952d-cbee90e21157 (0)' is not running after update. Review logs for failed jobs: web

Task 3042 Started  Tue Jun  6 11:34:16 UTC 2023
Task 3042 Finished Tue Jun  6 11:35:35 UTC 2023
Task 3042 Duration 00:01:19
Task 3042 error

Updating deployment:
  Expected task '3042' to succeed but state is 'error'

Exit code 1

web.stderr.log is as below

+ ENV_FILE_OWNER=vcap
+ mkdir -p /var/vcap/jobs/web/config/env
+ cat
+ [[ vcap != '' ]]
+ chown vcap:vcap /var/vcap/jobs/web/config/env/CONCOURSE_SESSION_SIGNING_KEY
+ chmod 0600 /var/vcap/jobs/web/config/env/CONCOURSE_SESSION_SIGNING_KEY
+ cat
+ [[ vcap != '' ]]
+ chown vcap:vcap /var/vcap/jobs/web/config/env/CONCOURSE_TSA_AUTHORIZED_KEYS
+ chmod 0600 /var/vcap/jobs/web/config/env/CONCOURSE_TSA_AUTHORIZED_KEYS
+ cat
+ [[ vcap != '' ]]
+ chown vcap:vcap /var/vcap/jobs/web/config/env/CONCOURSE_TSA_HOST_KEY
+ chmod 0600 /var/vcap/jobs/web/config/env/CONCOURSE_TSA_HOST_KEY
runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7fbcd6296a7c m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7fbcd6296a7c
stack: frame={sp:0x7ffc309c4550, fp:0x0} stack=[0x7ffc301c5aa8,0x7ffc309c4ad0)
00007ffc309c4450:  0000000000000000  0000000000000000 
00007ffc309c4460:  0000000000000000  0000000000000000 
00007ffc309c4470:  0000000000000000  0000000000000000 
00007ffc309c4480:  0000000000000000  0000000000000000 
00007ffc309c4490:  0000000000000000  0000000000000000 
00007ffc309c44a0:  0000000000000000  0000000000000000 
00007ffc309c44b0:  0000000000000000  0000000000000000 
00007ffc309c44c0:  0000000000000000  0000000000000000 
00007ffc309c44d0:  0000000000000000  0000000000000000 
00007ffc309c44e0:  0000000000000000  0000000000000000 
00007ffc309c44f0:  0000000000000000  0000000000000000 
00007ffc309c4500:  0000000000000000  0000000000000000 
00007ffc309c4510:  0000000000000000  0000000000000000 
00007ffc309c4520:  0000000000000000  0000000000000000 
00007ffc309c4530:  0000000000000000  0000000000000000 
00007ffc309c4540:  0000000000000000  00007fbcd6296a6e 
00007ffc309c4550: <0000000000000000  0000000000000000 
00007ffc309c4560:  0000000000000000  0000000000000000 
00007ffc309c4570:  0000000000000000  0000000000000000 
00007ffc309c4580:  0000000000000000  0000000000000000 
00007ffc309c4590:  0000000000000000  0000000000000000 
00007ffc309c45a0:  0000000000000000  0000000000000000 
00007ffc309c45b0:  0000000000000000  0000000000000000 
00007ffc309c45c0:  0000000000000000  0000000000000000 
00007ffc309c45d0:  0000000000000000  f0399fb5b6eae500 
00007ffc309c45e0:  00007fbcd65c7b80  0000000000000006 
00007ffc309c45f0:  0000000004e922e0  0000000000000011 
00007ffc309c4600:  00000000025d4334  00007fbcd6242476 
00007ffc309c4610:  00007fbcd641ae90  00007fbcd62287f3 
00007ffc309c4620:  0000000000000020  0000000000000000 
00007ffc309c4630:  0000000000000000  0000000000000000 
00007ffc309c4640:  0000000000000000  0000000000000000 
runtime: unknown pc 0x7fbcd6296a7c
stack: frame={sp:0x7ffc309c4550, fp:0x0} stack=[0x7ffc301c5aa8,0x7ffc309c4ad0)
00007ffc309c4450:  0000000000000000  0000000000000000 
00007ffc309c4460:  0000000000000000  0000000000000000 
00007ffc309c4470:  0000000000000000  0000000000000000 
00007ffc309c4480:  0000000000000000  0000000000000000 
00007ffc309c4490:  0000000000000000  0000000000000000 
00007ffc309c44a0:  0000000000000000  0000000000000000 
00007ffc309c44b0:  0000000000000000  0000000000000000 
00007ffc309c44c0:  0000000000000000  0000000000000000 
00007ffc309c44d0:  0000000000000000  0000000000000000 
00007ffc309c44e0:  0000000000000000  0000000000000000 
00007ffc309c44f0:  0000000000000000  0000000000000000 
00007ffc309c4500:  0000000000000000  0000000000000000 
00007ffc309c4510:  0000000000000000  0000000000000000 
00007ffc309c4520:  0000000000000000  0000000000000000 
00007ffc309c4530:  0000000000000000  0000000000000000 
00007ffc309c4540:  0000000000000000  00007fbcd6296a6e 
00007ffc309c4550: <0000000000000000  0000000000000000 
00007ffc309c4560:  0000000000000000  0000000000000000 
00007ffc309c4570:  0000000000000000  0000000000000000 
00007ffc309c4580:  0000000000000000  0000000000000000 
00007ffc309c4590:  0000000000000000  0000000000000000 
00007ffc309c45a0:  0000000000000000  0000000000000000 
00007ffc309c45b0:  0000000000000000  0000000000000000 
00007ffc309c45c0:  0000000000000000  0000000000000000 
00007ffc309c45d0:  0000000000000000  f0399fb5b6eae500 
00007ffc309c45e0:  00007fbcd65c7b80  0000000000000006 
00007ffc309c45f0:  0000000004e922e0  0000000000000011 
00007ffc309c4600:  00000000025d4334  00007fbcd6242476 
00007ffc309c4610:  00007fbcd641ae90  00007fbcd62287f3 
00007ffc309c4620:  0000000000000020  0000000000000000 
00007ffc309c4630:  0000000000000000  0000000000000000 
00007ffc309c4640:  0000000000000000  0000000000000000 

goroutine 1 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc000060798 sp=0xc000060790 pc=0x472c70
runtime.main()
    /usr/local/go/src/runtime/proc.go:130 +0x63 fp=0xc0000607e0 sp=0xc000060798 pc=0x4469a3
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000607e8 sp=0xc0000607e0 pc=0x474d41

rax    0x0
rbx    0x7fbcd65c7b80
rcx    0x7fbcd6296a7c
rdx    0x6
rdi    0x1
rsi    0x1
rbp    0x1
rsp    0x7ffc309c4550
r8     0x7ffc309c4620
r9     0x7fffffff
r10    0x8
r11    0x246
r12    0x6
r13    0x16
r14    0x25d4334
r15    0x0
rip    0x7fbcd6296a7c
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
+ ENV_FILE_OWNER=vcap
+ mkdir -p /var/vcap/jobs/web/config/env
+ cat
+ [[ vcap != '' ]]
+ chown vcap:vcap /var/vcap/jobs/web/config/env/CONCOURSE_SESSION_SIGNING_KEY
+ chmod 0600 /var/vcap/jobs/web/config/env/CONCOURSE_SESSION_SIGNING_KEY
+ cat
+ [[ vcap != '' ]]
+ chown vcap:vcap /var/vcap/jobs/web/config/env/CONCOURSE_TSA_AUTHORIZED_KEYS
+ chmod 0600 /var/vcap/jobs/web/config/env/CONCOURSE_TSA_AUTHORIZED_KEYS
+ cat
+ [[ vcap != '' ]]
+ chown vcap:vcap /var/vcap/jobs/web/config/env/CONCOURSE_TSA_HOST_KEY
+ chmod 0600 /var/vcap/jobs/web/config/env/CONCOURSE_TSA_HOST_KEY
runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7f4954c96a7c m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f4954c96a7c
stack: frame={sp:0x7ffc641f56f0, fp:0x0} stack=[0x7ffc639f6c48,0x7ffc641f5c70)
00007ffc641f55f0:  0000000000000000  0000000000000000 
00007ffc641f5600:  0000000000000000  0000000000000000 
00007ffc641f5610:  0000000000000000  0000000000000000 
00007ffc641f5620:  0000000000000000  0000000000000000 
00007ffc641f5630:  0000000000000000  0000000000000000 
00007ffc641f5640:  0000000000000000  0000000000000000 
00007ffc641f5650:  0000000000000000  0000000000000000 
00007ffc641f5660:  0000000000000000  0000000000000000 
00007ffc641f5670:  0000000000000000  0000000000000000 
00007ffc641f5680:  0000000000000000  0000000000000000 
00007ffc641f5690:  0000000000000000  0000000000000000 
00007ffc641f56a0:  0000000000000000  0000000000000000 
00007ffc641f56b0:  0000000000000000  0000000000000000 
00007ffc641f56c0:  0000000000000000  0000000000000000 
00007ffc641f56d0:  0000000000000000  0000000000000000 
00007ffc641f56e0:  0000000000000000  00007f4954c96a6e 
00007ffc641f56f0: <0000000000000000  0000000000000000 
00007ffc641f5700:  0000000000000000  0000000000000000 
00007ffc641f5710:  0000000000000000  0000000000000000 
00007ffc641f5720:  0000000000000000  0000000000000000 
00007ffc641f5730:  0000000000000000  0000000000000000 
00007ffc641f5740:  0000000000000000  0000000000000000 
00007ffc641f5750:  0000000000000000  0000000000000000 
00007ffc641f5760:  0000000000000000  0000000000000000 
00007ffc641f5770:  0000000000000000  952ac85bff77bf00 
00007ffc641f5780:  00007f4954f71b80  0000000000000006 
00007ffc641f5790:  0000000004fe32e0  0000000000000011 
00007ffc641f57a0:  00000000025d4334  00007f4954c42476 
00007ffc641f57b0:  00007f4954e1ae90  00007f4954c287f3 
00007ffc641f57c0:  0000000000000020  0000000000000000 
00007ffc641f57d0:  0000000000000000  0000000000000000 
00007ffc641f57e0:  0000000000000000  0000000000000000 
runtime: unknown pc 0x7f4954c96a7c
stack: frame={sp:0x7ffc641f56f0, fp:0x0} stack=[0x7ffc639f6c48,0x7ffc641f5c70)
00007ffc641f55f0:  0000000000000000  0000000000000000 
00007ffc641f5600:  0000000000000000  0000000000000000 
00007ffc641f5610:  0000000000000000  0000000000000000 
00007ffc641f5620:  0000000000000000  0000000000000000 
00007ffc641f5630:  0000000000000000  0000000000000000 
00007ffc641f5640:  0000000000000000  0000000000000000 
00007ffc641f5650:  0000000000000000  0000000000000000 
00007ffc641f5660:  0000000000000000  0000000000000000 
00007ffc641f5670:  0000000000000000  0000000000000000 
00007ffc641f5680:  0000000000000000  0000000000000000 
00007ffc641f5690:  0000000000000000  0000000000000000 
00007ffc641f56a0:  0000000000000000  0000000000000000 
00007ffc641f56b0:  0000000000000000  0000000000000000 
00007ffc641f56c0:  0000000000000000  0000000000000000 
00007ffc641f56d0:  0000000000000000  0000000000000000 
00007ffc641f56e0:  0000000000000000  00007f4954c96a6e 
00007ffc641f56f0: <0000000000000000  0000000000000000 
00007ffc641f5700:  0000000000000000  0000000000000000 
00007ffc641f5710:  0000000000000000  0000000000000000 
00007ffc641f5720:  0000000000000000  0000000000000000 
00007ffc641f5730:  0000000000000000  0000000000000000 
00007ffc641f5740:  0000000000000000  0000000000000000 
00007ffc641f5750:  0000000000000000  0000000000000000 
00007ffc641f5760:  0000000000000000  0000000000000000 
00007ffc641f5770:  0000000000000000  952ac85bff77bf00 
00007ffc641f5780:  00007f4954f71b80  0000000000000006 
00007ffc641f5790:  0000000004fe32e0  0000000000000011 
00007ffc641f57a0:  00000000025d4334  00007f4954c42476 
00007ffc641f57b0:  00007f4954e1ae90  00007f4954c287f3 
00007ffc641f57c0:  0000000000000020  0000000000000000 
00007ffc641f57d0:  0000000000000000  0000000000000000 
00007ffc641f57e0:  0000000000000000  0000000000000000 

goroutine 1 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc000060798 sp=0xc000060790 pc=0x472c70
runtime.main()
    /usr/local/go/src/runtime/proc.go:130 +0x63 fp=0xc0000607e0 sp=0xc000060798 pc=0x4469a3
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000607e8 sp=0xc0000607e0 pc=0x474d41

rax    0x0
rbx    0x7f4954f71b80
rcx    0x7f4954c96a7c
rdx    0x6
rdi    0x1
rsi    0x1
rbp    0x1
rsp    0x7ffc641f56f0
r8     0x7ffc641f57c0
r9     0x7fffffff
r10    0x8
r11    0x246
r12    0x6
r13    0x16
r14    0x25d4334
r15    0x0
rip    0x7f4954c96a7c
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
+ ENV_FILE_OWNER=vcap
+ mkdir -p /var/vcap/jobs/web/config/env
+ cat
+ [[ vcap != '' ]]
+ chown vcap:vcap /var/vcap/jobs/web/config/env/CONCOURSE_SESSION_SIGNING_KEY
+ chmod 0600 /var/vcap/jobs/web/config/env/CONCOURSE_SESSION_SIGNING_KEY
+ cat
+ [[ vcap != '' ]]
+ chown vcap:vcap /var/vcap/jobs/web/config/env/CONCOURSE_TSA_AUTHORIZED_KEYS
+ chmod 0600 /var/vcap/jobs/web/config/env/CONCOURSE_TSA_AUTHORIZED_KEYS
+ cat
+ [[ vcap != '' ]]
+ chown vcap:vcap /var/vcap/jobs/web/config/env/CONCOURSE_TSA_HOST_KEY
+ chmod 0600 /var/vcap/jobs/web/config/env/CONCOURSE_TSA_HOST_KEY
runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7f474f896a7c m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f474f896a7c
stack: frame={sp:0x7ffd33cfe5e0, fp:0x0} stack=[0x7ffd334ffb38,0x7ffd33cfeb60)
00007ffd33cfe4e0:  0000000000000000  0000000000000000 
00007ffd33cfe4f0:  0000000000000000  0000000000000000 
00007ffd33cfe500:  0000000000000000  0000000000000000 
00007ffd33cfe510:  0000000000000000  0000000000000000 
00007ffd33cfe520:  0000000000000000  0000000000000000 
00007ffd33cfe530:  0000000000000000  0000000000000000 
00007ffd33cfe540:  0000000000000000  0000000000000000 
00007ffd33cfe550:  0000000000000000  0000000000000000 
00007ffd33cfe560:  0000000000000000  0000000000000000 
00007ffd33cfe570:  0000000000000000  0000000000000000 
00007ffd33cfe580:  0000000000000000  0000000000000000 
00007ffd33cfe590:  0000000000000000  0000000000000000 
00007ffd33cfe5a0:  0000000000000000  0000000000000000 
00007ffd33cfe5b0:  0000000000000000  0000000000000000 
00007ffd33cfe5c0:  0000000000000000  0000000000000000 
00007ffd33cfe5d0:  0000000000000000  00007f474f896a6e 
00007ffd33cfe5e0: <0000000000000000  0000000000000000 
00007ffd33cfe5f0:  0000000000000000  0000000000000000 
00007ffd33cfe600:  0000000000000000  0000000000000000 
00007ffd33cfe610:  0000000000000000  0000000000000000 
00007ffd33cfe620:  0000000000000000  0000000000000000 
00007ffd33cfe630:  0000000000000000  0000000000000000 
00007ffd33cfe640:  0000000000000000  0000000000000000 
00007ffd33cfe650:  0000000000000000  0000000000000000 
00007ffd33cfe660:  0000000000000000  90d6c897e465aa00 
00007ffd33cfe670:  00007f474fb5eb80  0000000000000006 
00007ffd33cfe680:  000000000392b2e0  0000000000000011 
00007ffd33cfe690:  00000000025d4334  00007f474f842476 
00007ffd33cfe6a0:  00007f474fa1ae90  00007f474f8287f3 
00007ffd33cfe6b0:  0000000000000020  0000000000000000 
00007ffd33cfe6c0:  0000000000000000  0000000000000000 
00007ffd33cfe6d0:  0000000000000000  0000000000000000 
runtime: unknown pc 0x7f474f896a7c
stack: frame={sp:0x7ffd33cfe5e0, fp:0x0} stack=[0x7ffd334ffb38,0x7ffd33cfeb60)
00007ffd33cfe4e0:  0000000000000000  0000000000000000 
00007ffd33cfe4f0:  0000000000000000  0000000000000000 
00007ffd33cfe500:  0000000000000000  0000000000000000 
00007ffd33cfe510:  0000000000000000  0000000000000000 
00007ffd33cfe520:  0000000000000000  0000000000000000 
00007ffd33cfe530:  0000000000000000  0000000000000000 
00007ffd33cfe540:  0000000000000000  0000000000000000 
00007ffd33cfe550:  0000000000000000  0000000000000000 
00007ffd33cfe560:  0000000000000000  0000000000000000 
00007ffd33cfe570:  0000000000000000  0000000000000000 
00007ffd33cfe580:  0000000000000000  0000000000000000 
00007ffd33cfe590:  0000000000000000  0000000000000000 
00007ffd33cfe5a0:  0000000000000000  0000000000000000 
00007ffd33cfe5b0:  0000000000000000  0000000000000000 
00007ffd33cfe5c0:  0000000000000000  0000000000000000 
00007ffd33cfe5d0:  0000000000000000  00007f474f896a6e 
00007ffd33cfe5e0: <0000000000000000  0000000000000000 
00007ffd33cfe5f0:  0000000000000000  0000000000000000 
00007ffd33cfe600:  0000000000000000  0000000000000000 
00007ffd33cfe610:  0000000000000000  0000000000000000 
00007ffd33cfe620:  0000000000000000  0000000000000000 
00007ffd33cfe630:  0000000000000000  0000000000000000 
00007ffd33cfe640:  0000000000000000  0000000000000000 
00007ffd33cfe650:  0000000000000000  0000000000000000 
00007ffd33cfe660:  0000000000000000  90d6c897e465aa00 
00007ffd33cfe670:  00007f474fb5eb80  0000000000000006 
00007ffd33cfe680:  000000000392b2e0  0000000000000011 
00007ffd33cfe690:  00000000025d4334  00007f474f842476 
00007ffd33cfe6a0:  00007f474fa1ae90  00007f474f8287f3 
00007ffd33cfe6b0:  0000000000000020  0000000000000000 
00007ffd33cfe6c0:  0000000000000000  0000000000000000 
00007ffd33cfe6d0:  0000000000000000  0000000000000000

Reproduction steps

  1. bosh and cf deployed on aws (bosh version 277.3.2)
  2. try deploying concourse with following manifest
bosh -e $BOSH_ENVIRONMENT int -d concourse ${UPSTREAM_PATH}/cluster/concourse.yml \
  -l versions.yml \
  --vars-file vars.yml \
  --var azs=[z1] \
  -o ${UPSTREAM_PATH}/cluster/operations/basic-auth.yml

Expected behavior

Expects web instance to be active state with bosh

Additional context

No response

eyett commented 1 year ago

This issue is likely related with the version of bpm. In my case the version of bpm was overwritten with the older.