sensepost / gowitness

🔍 gowitness - a golang, web screenshot utility using Chrome Headless
GNU General Public License v3.0
3.16k stars 345 forks source link

nil pointer dereference in buntdb #34

Closed leonjza closed 4 years ago

leonjza commented 4 years ago

Latest version with the same problem when hitting pages that are returning just a JSON string (for example).

(...)
ERRO[2020-01-25 12:57:55] Screenshot failed                             destination=https-REDACTED.com.png err="exit status 1" url="https://REDACTED/"
panic: runtime error: invalid memory address or nil pointer dereference
(...)

The page returned this when using curl:

> GET / HTTP/1.1
> Host: REDACTED.com
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json
< Client-IP: REDACTED
< GatewayMessage: error
< WWW-Authenticate: Bearer
< X-Backside-Transport: FAIL FAIL
< Connection: close
<
* TLSv1.2 (IN), TLS alert, close notify (256):
* Closing connection 1
* TLSv1.2 (OUT), TLS alert, close notify (256):
{ "ServiceStatus":{ "StatusType":"Error", "ServiceCode":"", "ServiceName":"
(REDACTED)
 "InnerException":"", "ServerName":"" }

Originally posted by @BBerastegui in https://github.com/sensepost/gowitness/issues/24#issuecomment-578404476

leonjza commented 4 years ago

Trying to replicate this now, but I am not winning. Could you give this a bash @BBerastegui, or maybe tell me what I am missing?

package main

import (
    "encoding/json"
    "log"
    "net/http"
)

type person struct {
    Name string `json:"name"`
    Age  int    `json:"age"`
}

var tom *person = &person{
    Name: "Tom",
    Age:  28,
}

func tomHandler(w http.ResponseWriter, r *http.Request) {

    j, _ := json.Marshal(tom)
    w.Header().Add("Content-Type", "application/json")
    w.WriteHeader(http.StatusUnauthorized)
    w.Write(j)
}

func main() {
    http.HandleFunc("/", tomHandler)

    log.Println("Go!")
    http.ListenAndServe(":8080", nil)
}

Running it, I get the following from curl (trying to replicate the original request), with gowitness not crashing on me:

$ curl -v localhost:8080
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json
< Date: Thu, 30 Jan 2020 14:15:54 GMT
< Content-Length: 23
<
* Connection #0 to host localhost left intact
{"name":"Tom","age":28}* Closing connection 0
$ gowitness single --url=http://localhost:8080
INFO[2020-01-30 16:16:50] Title parsed                                  title= url="http://localhost:8080"
INFO[2020-01-30 16:16:50] Response code                                 status="401 Unauthorized" url="http://localhost:8080"
INFO[2020-01-30 16:16:50] Final URL after redirects                     final-url="http://localhost:8080" url="http://localhost:8080"
INFO[2020-01-30 16:16:50] Response header                               Date="Thu, 30 Jan 2020 14:16:50 GMT" url="http://localhost:8080"
INFO[2020-01-30 16:16:50] Response header                               Content-Length=23 url="http://localhost:8080"
INFO[2020-01-30 16:16:50] Response header                               Retry-Count=0 url="http://localhost:8080"
INFO[2020-01-30 16:16:50] Response header                               Content-Type=application/json url="http://localhost:8080"
INFO[2020-01-30 16:16:50] [--headless --disable-gpu --hide-scrollbars --disable-crash-reporter --user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.75 Safari/537.36 --window-size=1440,900 --screenshot=http-localhost-8080.png --virtual-time-budget=2000]
INFO[2020-01-30 16:16:50] Taking screenshot                             destination=http-localhost-8080.png url="http://localhost:8080"
INFO[2020-01-30 16:16:51] Screenshot taken                              destination=http-localhost-8080.png duration=1.06122262s url="http://localhost:8080"
INFO[2020-01-30 16:16:51] Complete
BBerastegui commented 4 years ago

I'm experiencing not only with JSON websites but with some others such as:

 $ docker run --rm -it leonjza/gowitness:latest single -u http://aasp-oqa.ual.com
INFO[2020-02-03 16:21:53] Title parsed                                  title="United Intranet Login" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response code                                 status="200 OK" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Final URL after redirects                     final-url="https://login-qa.ual.com/oamsso-bin/login.pl?contextType=external&username=string&OverrideRetryLimit=0&password=secure_string&challenge_url=https%3A%2F%2Flogin-qa.ual.com%2Foamsso-bin%2Flogin.pl&creds=userid+password&request_id=5666347639047595962&authn_try_count=0&locale=en_US&resource_url=https%253A%252F%252Faasp-oqa.ual.com%252F" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header                               Date="Mon, 03 Feb 2020 16:18:20 GMT" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header                               Cache-Control="no-cache, no-store" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header                               Access-Control-Allow-Credentials=true url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header                               X-Frame-Options=DENY url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header                               Strict-Transport-Security="max-age=31536000; preload" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header                               Retry-Count=0 url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header                               Server=Oracle-HTTP-Server-11g url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header                               Expires=0 url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header                               Pragma=no-cache url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header                               Generated-By=login.pl url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header                               Content-Type="text/html; charset=UTF-8" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header                               Content-Language=en url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Certificate chain common name                 common_name="*.ual.com" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Signature algorithm                           signature-alg=SHA256-RSA url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Public key algorithm                          pubkey-alg=RSA url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Issuer                                        issuer="GeoTrust TLS RSA CA G1" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] DNS Name                                      dns-names="*.ual.com" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] DNS Name                                      dns-names=ual.com url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Certificate chain common name                 common_name="GeoTrust TLS RSA CA G1" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Signature algorithm                           signature-alg=SHA256-RSA url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Public key algorithm                          pubkey-alg=RSA url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Issuer                                        issuer="DigiCert Global Root G2" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Cipher suite in use                           cipher-suite=49200 url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] [--headless --disable-gpu --hide-scrollbars --disable-crash-reporter --user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.75 Safari/537.36 --window-size=1440,900 --screenshot=http-aasp-oqa.ual.com.png --virtual-time-budget=2000 -no-sandbox]
INFO[2020-02-03 16:21:53] Taking screenshot                             destination=http-aasp-oqa.ual.com.png url="https://login-qa.ual.com/?contextType=external&username=string&OverrideRetryLimit=0&password=secure_string&challenge_url=https%3A%2F%2Flogin-qa.ual.com%2Foamsso-bin%2Flogin.pl&creds=userid+password&request_id=5666347639047595962&authn_try_count=0&locale=en_US&resource_url=https%253A%252F%252Faasp-oqa.ual.com%252F"

ERRO[2020-02-03 16:23:23] Timeout reached while waiting for screenshot to finish  destination=http-aasp-oqa.ual.com.png err="signal: killed" url="https://login-qa.ual.com/?contextType=external&username=string&OverrideRetryLimit=0&password=secure_string&challenge_url=https%3A%2F%2Flogin-qa.ual.com%2Foamsso-bin%2Flogin.pl&creds=userid+password&request_id=5666347639047595962&authn_try_count=0&locale=en_US&resource_url=https%253A%252F%252Faasp-oqa.ual.com%252F"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7fe535]

goroutine 1 [running]:
github.com/tidwall/buntdb.(*Tx).lock(0xc0003f85a0)
    /go/pkg/mod/github.com/tidwall/buntdb@v0.0.0-20170824144000-b67b1b8c1658/buntdb.go:1068 +0x65
github.com/tidwall/buntdb.(*DB).Begin(0x0, 0xc000182301, 0x3, 0xc000432800, 0x5cbc57)
    /go/pkg/mod/github.com/tidwall/buntdb@v0.0.0-20170824144000-b67b1b8c1658/buntdb.go:1047 +0x61
github.com/tidwall/buntdb.(*DB).managed(0x0, 0xc000432901, 0xc000432920, 0x0, 0x0)
    /go/pkg/mod/github.com/tidwall/buntdb@v0.0.0-20170824144000-b67b1b8c1658/buntdb.go:919 +0x57
github.com/tidwall/buntdb.(*DB).Update(...)
    /go/pkg/mod/github.com/tidwall/buntdb@v0.0.0-20170824144000-b67b1b8c1658/buntdb.go:964
github.com/sensepost/gowitness/storage.(*Storage).SetHTTPData(0xf5af30, 0xc0001bc460)
    /src/storage/storage.go:64 +0x506
github.com/sensepost/gowitness/utils.ProcessURL(0xc0000c6c00, 0xf5d0e0, 0xf5af30, 0x3)
    /src/utils/processor.go:115 +0x20bf
github.com/sensepost/gowitness/cmd.glob..func7(0xf53780, 0xc000180bd0, 0x0, 0x3)
    /src/cmd/single.go:36 +0x8c
github.com/spf13/cobra.(*Command).execute(0xf53780, 0xc000180b40, 0x3, 0x3, 0xf53780, 0xc000180b40)
    /go/pkg/mod/github.com/spf13/cobra@v0.0.1/command.go:702 +0x285
github.com/spf13/cobra.(*Command).ExecuteC(0xf53340, 0xc000000180, 0xc000179f50, 0x4072ff)
    /go/pkg/mod/github.com/spf13/cobra@v0.0.1/command.go:783 +0x2c9
github.com/spf13/cobra.(*Command).Execute(...)
    /go/pkg/mod/github.com/spf13/cobra@v0.0.1/command.go:736
github.com/sensepost/gowitness/cmd.Execute()
    /src/cmd/root.go:122 +0x31
main.main()
    /src/main.go:6 +0x20
leonjza commented 4 years ago

Ah that is useful and gives me a hint towards what may be going on.

leonjza commented 4 years ago

V2 uses sqlite now.