Closed leonjza closed 4 years ago
Trying to replicate this now, but I am not winning. Could you give this a bash @BBerastegui, or maybe tell me what I am missing?
package main
import (
"encoding/json"
"log"
"net/http"
)
type person struct {
Name string `json:"name"`
Age int `json:"age"`
}
var tom *person = &person{
Name: "Tom",
Age: 28,
}
func tomHandler(w http.ResponseWriter, r *http.Request) {
j, _ := json.Marshal(tom)
w.Header().Add("Content-Type", "application/json")
w.WriteHeader(http.StatusUnauthorized)
w.Write(j)
}
func main() {
http.HandleFunc("/", tomHandler)
log.Println("Go!")
http.ListenAndServe(":8080", nil)
}
Running it, I get the following from curl (trying to replicate the original request), with gowitness not crashing on me:
$ curl -v localhost:8080
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json
< Date: Thu, 30 Jan 2020 14:15:54 GMT
< Content-Length: 23
<
* Connection #0 to host localhost left intact
{"name":"Tom","age":28}* Closing connection 0
$ gowitness single --url=http://localhost:8080
INFO[2020-01-30 16:16:50] Title parsed title= url="http://localhost:8080"
INFO[2020-01-30 16:16:50] Response code status="401 Unauthorized" url="http://localhost:8080"
INFO[2020-01-30 16:16:50] Final URL after redirects final-url="http://localhost:8080" url="http://localhost:8080"
INFO[2020-01-30 16:16:50] Response header Date="Thu, 30 Jan 2020 14:16:50 GMT" url="http://localhost:8080"
INFO[2020-01-30 16:16:50] Response header Content-Length=23 url="http://localhost:8080"
INFO[2020-01-30 16:16:50] Response header Retry-Count=0 url="http://localhost:8080"
INFO[2020-01-30 16:16:50] Response header Content-Type=application/json url="http://localhost:8080"
INFO[2020-01-30 16:16:50] [--headless --disable-gpu --hide-scrollbars --disable-crash-reporter --user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.75 Safari/537.36 --window-size=1440,900 --screenshot=http-localhost-8080.png --virtual-time-budget=2000]
INFO[2020-01-30 16:16:50] Taking screenshot destination=http-localhost-8080.png url="http://localhost:8080"
INFO[2020-01-30 16:16:51] Screenshot taken destination=http-localhost-8080.png duration=1.06122262s url="http://localhost:8080"
INFO[2020-01-30 16:16:51] Complete
I'm experiencing not only with JSON websites but with some others such as:
$ docker run --rm -it leonjza/gowitness:latest single -u http://aasp-oqa.ual.com
INFO[2020-02-03 16:21:53] Title parsed title="United Intranet Login" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response code status="200 OK" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Final URL after redirects final-url="https://login-qa.ual.com/oamsso-bin/login.pl?contextType=external&username=string&OverrideRetryLimit=0&password=secure_string&challenge_url=https%3A%2F%2Flogin-qa.ual.com%2Foamsso-bin%2Flogin.pl&creds=userid+password&request_id=5666347639047595962&authn_try_count=0&locale=en_US&resource_url=https%253A%252F%252Faasp-oqa.ual.com%252F" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header Date="Mon, 03 Feb 2020 16:18:20 GMT" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header Cache-Control="no-cache, no-store" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header Access-Control-Allow-Credentials=true url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header X-Frame-Options=DENY url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header Strict-Transport-Security="max-age=31536000; preload" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header Retry-Count=0 url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header Server=Oracle-HTTP-Server-11g url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header Expires=0 url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header Pragma=no-cache url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header Generated-By=login.pl url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header Content-Type="text/html; charset=UTF-8" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Response header Content-Language=en url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Certificate chain common name common_name="*.ual.com" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Signature algorithm signature-alg=SHA256-RSA url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Public key algorithm pubkey-alg=RSA url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Issuer issuer="GeoTrust TLS RSA CA G1" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] DNS Name dns-names="*.ual.com" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] DNS Name dns-names=ual.com url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Certificate chain common name common_name="GeoTrust TLS RSA CA G1" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Signature algorithm signature-alg=SHA256-RSA url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Public key algorithm pubkey-alg=RSA url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Issuer issuer="DigiCert Global Root G2" url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] Cipher suite in use cipher-suite=49200 url="http://aasp-oqa.ual.com"
INFO[2020-02-03 16:21:53] [--headless --disable-gpu --hide-scrollbars --disable-crash-reporter --user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.75 Safari/537.36 --window-size=1440,900 --screenshot=http-aasp-oqa.ual.com.png --virtual-time-budget=2000 -no-sandbox]
INFO[2020-02-03 16:21:53] Taking screenshot destination=http-aasp-oqa.ual.com.png url="https://login-qa.ual.com/?contextType=external&username=string&OverrideRetryLimit=0&password=secure_string&challenge_url=https%3A%2F%2Flogin-qa.ual.com%2Foamsso-bin%2Flogin.pl&creds=userid+password&request_id=5666347639047595962&authn_try_count=0&locale=en_US&resource_url=https%253A%252F%252Faasp-oqa.ual.com%252F"
ERRO[2020-02-03 16:23:23] Timeout reached while waiting for screenshot to finish destination=http-aasp-oqa.ual.com.png err="signal: killed" url="https://login-qa.ual.com/?contextType=external&username=string&OverrideRetryLimit=0&password=secure_string&challenge_url=https%3A%2F%2Flogin-qa.ual.com%2Foamsso-bin%2Flogin.pl&creds=userid+password&request_id=5666347639047595962&authn_try_count=0&locale=en_US&resource_url=https%253A%252F%252Faasp-oqa.ual.com%252F"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7fe535]
goroutine 1 [running]:
github.com/tidwall/buntdb.(*Tx).lock(0xc0003f85a0)
/go/pkg/mod/github.com/tidwall/buntdb@v0.0.0-20170824144000-b67b1b8c1658/buntdb.go:1068 +0x65
github.com/tidwall/buntdb.(*DB).Begin(0x0, 0xc000182301, 0x3, 0xc000432800, 0x5cbc57)
/go/pkg/mod/github.com/tidwall/buntdb@v0.0.0-20170824144000-b67b1b8c1658/buntdb.go:1047 +0x61
github.com/tidwall/buntdb.(*DB).managed(0x0, 0xc000432901, 0xc000432920, 0x0, 0x0)
/go/pkg/mod/github.com/tidwall/buntdb@v0.0.0-20170824144000-b67b1b8c1658/buntdb.go:919 +0x57
github.com/tidwall/buntdb.(*DB).Update(...)
/go/pkg/mod/github.com/tidwall/buntdb@v0.0.0-20170824144000-b67b1b8c1658/buntdb.go:964
github.com/sensepost/gowitness/storage.(*Storage).SetHTTPData(0xf5af30, 0xc0001bc460)
/src/storage/storage.go:64 +0x506
github.com/sensepost/gowitness/utils.ProcessURL(0xc0000c6c00, 0xf5d0e0, 0xf5af30, 0x3)
/src/utils/processor.go:115 +0x20bf
github.com/sensepost/gowitness/cmd.glob..func7(0xf53780, 0xc000180bd0, 0x0, 0x3)
/src/cmd/single.go:36 +0x8c
github.com/spf13/cobra.(*Command).execute(0xf53780, 0xc000180b40, 0x3, 0x3, 0xf53780, 0xc000180b40)
/go/pkg/mod/github.com/spf13/cobra@v0.0.1/command.go:702 +0x285
github.com/spf13/cobra.(*Command).ExecuteC(0xf53340, 0xc000000180, 0xc000179f50, 0x4072ff)
/go/pkg/mod/github.com/spf13/cobra@v0.0.1/command.go:783 +0x2c9
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/cobra@v0.0.1/command.go:736
github.com/sensepost/gowitness/cmd.Execute()
/src/cmd/root.go:122 +0x31
main.main()
/src/main.go:6 +0x20
Ah that is useful and gives me a hint towards what may be going on.
V2 uses sqlite now.
Latest version with the same problem when hitting pages that are returning just a JSON string (for example).
The page returned this when using curl:
Originally posted by @BBerastegui in https://github.com/sensepost/gowitness/issues/24#issuecomment-578404476