search

devspace-sh / devspace

DevSpace - The Fastest Developer Tool for Kubernetes ⚡ Automate your deployment workflow with DevSpace and develop software directly inside Kubernetes.
https://devspace.sh
Apache License 2.0
4.22k stars 353 forks source link

fatal: bad g in signal handler #1334

Closed Orgil closed 1 year ago

Orgil commented 3 years ago

What happened?
devspace dev command raises error and restarted again. Each restart giving different following errors.

bad g in signal handler, morestack on g0 signal SIGSEGV: segmentation violation code=0x1 addr=0x720 pc=0x8087e1a

What did you expect to happen instead?
it previously worked fine on version 5.7.4, recently i reinstalled my OS and installed devspace version 5.8.2 got error. Even i've tried with previous versions.

How can we reproduce the bug? (as minimally and precisely as possible)

Local Environment:

Kubernetes Cluster:

Anything else we need to know?
is there any global dependencies that devspace need to run?

/kind bug

FabianKramm commented 3 years ago

@Orgil thanks for opening this issue! Could you post the complete devspace dev and the relevant devspace.yaml here, so that we can investigate this error?

Orgil commented 3 years ago

yaml file

version: v1beta9
images:
  master-app:
    image: master-app
    dockerfile: ./Dockerfile.dev
    preferSyncOverRebuild: true
    injectRestartHelper: false
    appendDockerfileInstructions:
    - USER root
    build:
      docker:
        skipPush: true
deployments:
- name: master-app
  helm:
    componentChart: true
    values:
      containers:
      - image: master-app
      labels:
        master-service: dev
dev:
  autoReload:
    paths:
    - ./package.json
    - ./Dockerfile.dev
    images:
    - master-app
    deployments:
    - master-app
  sync:
  - imageName: master-app
    disableDownload: true
    localSubPath: ./
    containerPath: /usr/app
    excludePaths:
    - .git/
    - chart/
    - .devspace/
    - .jest/
    - node_modules/
    - tests/
    - devspace.yaml
commands:
  - name: migrate
    command: "devspace enter yarn knex migrate:latest"
  - name: rollback
    command: "devspace enter yarn knex migrate:rollback"
  - name: seed
    command: "devspace enter yarn knex seed:run"

devspace output

[0:sync:master-app] Error: Sync Error on /Users/orgil/Projects/master-projects/master-app: Sync - connection lost to pod service-staging/master-app-545cfb76c4-pd7ct: fatal: bad g in signal handler
fatal: bad g in signal handler
 command terminated with exit code 2
[0:sync:master-app] Sync stopped
[0:sync:master-app] Restarting sync...
[0:sync:master-app] Start syncing
[0:sync:master-app] Sync started on /Users/orgil/Projects/master-projects/master-app <-> /usr/app (Pod: service-staging/master-app-545cfb76c4-pd7ct)
[0:sync:master-app] Error: Sync Error on /Users/orgil/Projects/master-projects/master-app: Sync - connection lost to pod service-staging/master-app-545cfb76c4-pd7ct: fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x720 pc=0x8087e1a]

runtime stack:
runtime.throw(0x84d04b8, 0x2a)
    /Users/runner/hostedtoolcache/go/1.16.0/x64/src/runtime/panic.go:1117 +0x6a
runtime.sigpanic()
    /Users/runner/hostedtoolcache/go/1.16.0/x64/src/runtime/signal_unix.go:718 +0x23f
runtime.runqput(0x0, 0x8c00380, 0x1)
    /Users/runner/hostedtoolcache/go/1.16.0/x64/src/runtime/proc.go:5746 +0x2a
runtime.newproc.func1()
    /Users/runner/hostedtoolcache/go/1.16.0/x64/src/runtime/proc.go:3982 +0x66
runtime.systemstack(0x80a6804)
    /Users/runner/hostedtoolcache/go/1.16.0/x64/src/runtime/asm_386.s:391 +0x53
runtime.mstart()
    /Users/runner/hostedtoolcache/go/1.16.0/x64/src/runtime/proc.go:1246

goroutine 1 [running, locked to thread]:
    goroutine running on other thread; stack unavailable
 command terminated with exit code 2
[0:sync:master-app] Sync stopped
[0:sync:master-app] Restarting sync...
[0:sync:master-app] Start syncing
[0:sync:master-app] Sync started on /Users/orgil/Projects/master-projects/master-app <-> /usr/app (Pod: service-staging/master-app-545cfb76c4-pd7ct)
[0:sync:master-app] Error: Sync Error on /Users/orgil/Projects/master-projects/master-app: Sync - connection lost to pod service-staging/master-app-545cfb76c4-pd7ct: fatal: bad g in signal handler
 command terminated with exit code 2
[0:sync:master-app] Sync stopped
[0:sync:master-app] Restarting sync...
Orgil commented 3 years ago

@FabianKramm As you can see above, it restarts when error occured. Happy to provide any additional info.

FabianKramm commented 3 years ago

@Orgil thanks for providing the additional info!

It seems the devspacehelper that is injected into the container does not work correctly because an unexpected operating system signal is interrupting it, which indicates that there might be something wrong with your kubernetes installation. Is the kubernetes installation besides that running correctly or are there any problems like DiskPressure for example? If not, you could test manually injecting the devspacehelper and check if you can execute it via kubectl:

  1. Download the devspacehelper binary from the releases page
  2. Copy the helper into the container via kubectl cp ./devspacehelper <some-pod>:/tmp/devspacehelper
  3. Check if you can execute it via kubectl exec -it <some-pod> -- /tmp/devspacehelper version
Orgil commented 3 years ago

@FabianKramm thanks for taking time. i've reinstalled kubernetes and run devspacehelper as you suggested. Still gives me same error.

❯ k exec -it master-app-585887f7cb-w6blp -- /tmp/devspacehelper
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x720 pc=0x8087e1a]

runtime stack:
runtime.throw(0x84d04b8, 0x2a)
    /Users/runner/hostedtoolcache/go/1.16.0/x64/src/runtime/panic.go:1117 +0x6a
runtime.sigpanic()
    /Users/runner/hostedtoolcache/go/1.16.0/x64/src/runtime/signal_unix.go:718 +0x23f
runtime.runqput(0x0, 0x8c00380, 0x1)
    /Users/runner/hostedtoolcache/go/1.16.0/x64/src/runtime/proc.go:5746 +0x2a
runtime.newproc.func1()
    /Users/runner/hostedtoolcache/go/1.16.0/x64/src/runtime/proc.go:3982 +0x66
runtime.systemstack(0x80a6804)
    /Users/runner/hostedtoolcache/go/1.16.0/x64/src/runtime/asm_386.s:391 +0x53
runtime.mstart()
    /Users/runner/hostedtoolcache/go/1.16.0/x64/src/runtime/proc.go:1246

goroutine 1 [running, locked to thread]:
    goroutine running on other thread; stack unavailable
command terminated with exit code 2
Orgil commented 3 years ago

btw, build image is node:14-alpine.

FabianKramm commented 3 years ago

@Orgil thanks for the info! I'll test it with the node image tomorrow. However, the error is really strange, is it possible that you run on ARM architecture or cannot run 32bit programs inside containers?

Orgil commented 3 years ago

@FabianKramm is it some how related to bigsur or latest update of docker? as my understanding, docker engine updated for arm architecture lately. my laptop is intel based old one.

Orgil commented 3 years ago

@FabianKramm I did some investigation on my end, so downgrading docker for mac to v3.1.0 works fine. It seems issue is related qemu upgrades on latest docker for mac v3.2.0.

FabianKramm commented 3 years ago

@Orgil thanks for the info with docker! I can reproduce the problem with docker v3.2.0 and get exactly the same error as you have. Seems like the problem is the 32bit devspacehelper. We'll upgrade it to 64bit for the next release which solved this problem on my installation!

FabianKramm commented 3 years ago

@Orgil I released v5.8.3-beta.0 which includes the fix, you can upgrade via devspace upgrade --version=v5.8.3-beta.0. Would be great if you could check if that fixes the issue

carlmontanari commented 1 year ago

Looks like this is hopefully all sorted out a while back, so we'll go ahead and close this one. Feel free to reopen if this issue pops back up!