search

projectdiscovery / pdtm

ProjectDiscovery's Open Source Tool Manager
https://projectdiscovery.io
MIT License
677 stars 49 forks source link

[Go 1.21.1] _cgo_pthread_key_created is missing on Ubuntu WSL #198

Closed bagasislamay closed 9 months ago

bagasislamay commented 10 months ago

pdtm version:

Latest

Current Behavior:

Error while running pdtm --help

striker@EUREKA:~$ go version
go version go1.21.1 linux/amd64
striker@EUREKA:~$ pdtm --help
fatal error: _cgo_pthread_key_created missing

goroutine 1 [running, locked to thread]:
runtime.throw({0xf0a07d?, 0x0?})
        /usr/local/go/src/runtime/panic.go:1077 +0x5c fp=0xc000086740 sp=0xc000086710 pc=0x438edc
runtime.main()
        /usr/local/go/src/runtime/proc.go:213 +0x409 fp=0xc0000867e0 sp=0xc000086740 pc=0x43ba09
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000867e8 sp=0xc0000867e0 pc=0x46b301

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000086fa8 sp=0xc000086f88 pc=0x43bd0e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:404
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:322 +0xb3 fp=0xc000086fe0 sp=0xc000086fa8 pc=0x43bb93
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000086fe8 sp=0xc000086fe0 pc=0x46b301
created by runtime.init.6 in goroutine 1
        /usr/local/go/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000087778 sp=0xc000087758 pc=0x43bd0e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
        /usr/local/go/src/runtime/mgcsweep.go:280 +0x94 fp=0xc0000877c8 sp=0xc000087778 pc=0x427c94
runtime.gcenable.func1()
        /usr/local/go/src/runtime/mgc.go:200 +0x25 fp=0xc0000877e0 sp=0xc0000877c8 pc=0x41ce25
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000877e8 sp=0xc0000877e0 pc=0x46b301
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc0000ae000?, 0x10fd6d8?, 0x1?, 0x0?, 0xc0000071e0?)
        /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc000087f70 sp=0xc000087f50 pc=0x43bd0e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x1729a60)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000087fa0 sp=0xc000087f70 pc=0x425529
runtime.bgscavenge(0x0?)
        /usr/local/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000087fc8 sp=0xc000087fa0 pc=0x425abc
runtime.gcenable.func2()
        /usr/local/go/src/runtime/mgc.go:201 +0x25 fp=0xc000087fe0 sp=0xc000087fc8 pc=0x41cdc5
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000087fe8 sp=0xc000087fe0 pc=0x46b301
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:201 +0xa5

Expected Behavior:

Can list out the content of pftm --help command

Steps To Reproduce:

  1. Install Go 1.21.1 in Ubuntu WSL, I follow the steps from https://go.dev/doc/install for Linux machines
  2. Install PDTM via go install -v github.com/projectdiscovery/pdtm/cmd/pdtm@latest
  3. Run pdtm --help
junanc commented 10 months ago

I also encountered the same issue.

matthewomccorkle commented 10 months ago

This is the same case for Ubuntu 22.04.3 LTS (jammy) from the AWS marketplace.

Mzack9999 commented 9 months ago

Steps to reproduce:

$ docker run -it --rm archlinux bash
[root@ef0ad1363234 /]# pacman -Syu
...
[root@ef0ad1363234 /]# pacman -S go
...
[root@ef0ad1363234 /]# go install -v github.com/projectdiscovery/pdtm/cmd/pdtm@latest
...
[root@ef0ad1363234 ~]# /root/go/bin/pdtm
fatal error: _cgo_pthread_key_created missing

goroutine 1 [running, locked to thread]:
runtime.throw({0xf0a07d?, 0x0?})
    /usr/lib/go/src/runtime/panic.go:1077 +0x5c fp=0xc000062740 sp=0xc000062710 pc=0x438edc
runtime.main()
    /usr/lib/go/src/runtime/proc.go:213 +0x409 fp=0xc0000627e0 sp=0xc000062740 pc=0x43ba09
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000627e8 sp=0xc0000627e0 pc=0x46b301

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/lib/go/src/runtime/proc.go:398 +0xce fp=0xc000062fa8 sp=0xc000062f88 pc=0x43bd0e
runtime.goparkunlock(...)
    /usr/lib/go/src/runtime/proc.go:404
runtime.forcegchelper()
    /usr/lib/go/src/runtime/proc.go:322 +0xb3 fp=0xc000062fe0 sp=0xc000062fa8 pc=0x43bb93
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000062fe8 sp=0xc000062fe0 pc=0x46b301
created by runtime.init.6 in goroutine 1
    /usr/lib/go/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
    /usr/lib/go/src/runtime/proc.go:398 +0xce fp=0xc000063778 sp=0xc000063758 pc=0x43bd0e
runtime.goparkunlock(...)
    /usr/lib/go/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
    /usr/lib/go/src/runtime/mgcsweep.go:280 +0x94 fp=0xc0000637c8 sp=0xc000063778 pc=0x427c94
runtime.gcenable.func1()
    /usr/lib/go/src/runtime/mgc.go:200 +0x25 fp=0xc0000637e0 sp=0xc0000637c8 pc=0x41ce25
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0000637e8 sp=0xc0000637e0 pc=0x46b301
created by runtime.gcenable in goroutine 1
    /usr/lib/go/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xc00007a000?, 0x10fd6d8?, 0x1?, 0x0?, 0xc0000071e0?)
    /usr/lib/go/src/runtime/proc.go:398 +0xce fp=0xc000063f70 sp=0xc000063f50 pc=0x43bd0e
runtime.goparkunlock(...)
    /usr/lib/go/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x1727a60)
    /usr/lib/go/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc000063fa0 sp=0xc000063f70 pc=0x425529
runtime.bgscavenge(0x0?)
    /usr/lib/go/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc000063fc8 sp=0xc000063fa0 pc=0x425abc
runtime.gcenable.func2()
    /usr/lib/go/src/runtime/mgc.go:201 +0x25 fp=0xc000063fe0 sp=0xc000063fc8 pc=0x41cdc5
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc000063fe8 sp=0xc000063fe0 pc=0x46b301
created by runtime.gcenable in goroutine 1
    /usr/lib/go/src/runtime/mgc.go:201 +0xa5
Mzack9999 commented 9 months ago

Looks like this can be fixed with installing gcc on the linux distribution so that pthread is automatically added to the compilation flags:

$ go env
...
GOGCCFLAGS='-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2242934390=/tmp/go-build -gno-record-gcc-switches'
$ pacman -S gcc
$ go env
...
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3322364864=/tmp/go-build -gno-record-gcc-switches'
bagasislamay commented 9 months ago

Hey @Mzack9999 I have gcc installed and go env listed out similar GOGCCFLAGS value, but still facing the same issue. With gcc installed on your end, can you run any pdtm command?

Mzack9999 commented 9 months ago

@bagasislamay yeah, by installing gcc everything worked for me. Could you provide more info about the host operating system. I've tested it with ubuntu 23.10 64bit running arch on docker

bagasislamay commented 9 months ago

Hey @Mzack9999 I am on Ubuntu 22.04.3 LTS running on WSL with gcc installed. I have no idea about OS versioning and package management stuff and still couldn't find any way around to solve this issue up until this day

Mzack9999 commented 9 months ago

@bagasislamay Ubuntu 22.04.xx LTS on WSL usually has a very old go version (go1.13), I was able to run pdtm successfully on windows 11 with WSL and manual installation of go 1.21.4. What is the host OS?

Also, could you try to locate the position of libpthread.so, usually:

$ ls /usr/lib/x86_64-linux-gnu/ | grep thread
bagasislamay commented 9 months ago

Hey @Mzack9999, thanks to you the issue is now resolved. For those who still face the issue:

  1. I updated Go from version 1.21.1 to latest version of 1.21.3 on my machine: sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.21.3.linux-amd64.tar.gz
  2. Then I updated PDTM: go install -v github.com/projectdiscovery/pdtm/cmd/pdtm@latest so my current PDTM version is v0.0.9

Then PDTM is now working properly and also I will close this issue😊