Trouble with --iterative

[theloglizard]

I can do a clone but can't seem to do a checkout as advised. I see there were some recent edits, there, so I'm hoping there's some other updates that maybe got another wire crossed.

Debian:~$ git clone https://github.com/zmap/zdns.git
Cloning into 'zdns'...
remote: Enumerating objects: 78, done.
remote: Counting objects: 100% (78/78), done.
remote: Compressing objects: 100% (51/51), done.
remote: Total 1684 (delta 33), reused 42 (delta 15), pack-reused 1606
Receiving objects: 100% (1684/1684), 382.38 KiB | 4.66 MiB/s, done.
Resolving deltas: 100% (930/930), done.
Debian:~$ cd zdns
Debian:~/zdns$ go build
go: finding github.com/hashicorp/go-version v1.2.0
go: finding github.com/liip/sheriff v0.0.0-20190308094614-91aa83a45a3d
go: finding github.com/zmap/dns v1.1.28-zmap-3
go: finding github.com/kavu/go_reuseport v1.4.0
go: finding github.com/asergeyev/nradix v0.0.0-20170505151046-3872ab85bb56
go: finding github.com/sirupsen/logrus v1.4.2
go: finding github.com/zmap/go-iptree v0.0.0-20170831022036-1948b1097e25
go: finding golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa
go: finding golang.org/x/crypto v0.0.0-20200117160349-530e935923ad
go: finding golang.org/x/net v0.0.0-20190923162816-aa69164e4478
go: finding golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550
go: finding github.com/stretchr/testify v1.2.2
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding golang.org/x/sync v0.0.0-20190423024810-112230192c58
go: finding golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe
go: finding golang.org/x/sys v0.0.0-20190422165155-953cdadca894
go: finding golang.org/x/tools v0.0.0-20191216052735-49a3e744a425
go: finding github.com/stretchr/objx v0.1.1
go: finding github.com/davecgh/go-spew v1.1.1
go: finding golang.org/x/sys v0.0.0-20190412213103-97732733099d
go: finding github.com/konsorten/go-windows-terminal-sequences v1.0.1
go: finding golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3
go: finding golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a
go: finding golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2
go: finding golang.org/x/text v0.3.0
go: finding golang.org/x/net v0.0.0-20190620200207-3b0461eec859
go: finding golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee
go: finding golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898
go: downloading github.com/zmap/dns v1.1.28-zmap-3
go: downloading github.com/sirupsen/logrus v1.4.2
go: downloading github.com/liip/sheriff v0.0.0-20190308094614-91aa83a45a3d
go: downloading github.com/hashicorp/go-version v1.2.0
go: downloading golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe
go: downloading golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa
go: downloading golang.org/x/crypto v0.0.0-20200117160349-530e935923ad
go: finding github.com/miekg/dns v1.1.27

_Then a make install:_

Debian:~/zdns$ make install
cd zdns && go build
go: downloading github.com/zmap/go-iptree v0.0.0-20170831022036-1948b1097e25
go: downloading github.com/asergeyev/nradix v0.0.0-20170505151046-3872ab85bb56
cd zdns && go install

_I get something to work:_

echo "censys.io" | zdns A
{"data":{"answers":[{"answer":"104.26.10.245","class":"IN","name":"censys.io","ttl":300,"type":"A"},{"answer":"104.26.11.245","class":"IN","name":"censys.io","ttl":300,"type":"A"}],"protocol":"udp","resolver":"192.168.0.1:53"},"name":"censys.io","status":"NOERROR","timestamp":"2020-05-23T16:59:51-04:00"}

#########################################################
_But the iterative mode seems to crash, even on a simple example_

Debian:~/zdns$ echo "censys.io" | zdns A --iterative
panic: runtime error: index out of range

goroutine 21 [running]:
github.com/zmap/zdns/cachehash.(*ShardedCacheHash).getShard(0x9c90394, 0x8280920, 0x9d80090, 0x9d80002)
    /home/user/zdns/cachehash/shardedcachehash.go:42 +0x54
github.com/zmap/zdns/cachehash.(*ShardedCacheHash).Lock(0x9c90394, 0x8280920, 0x9d80090)
    /home/user/zdns/cachehash/shardedcachehash.go:72 +0x31
github.com/zmap/zdns/modules/miekg.(*GlobalLookupFactory).GetCachedResult(0x9c90390, 0x9d5a007, 0x2, 0x10002, 0x3, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/user/zdns/modules/miekg/miekg.go:194 +0x1c8
github.com/zmap/zdns/modules/miekg.(*Lookup).cachedRetryingLookup(0x9d6e030, 0x10001, 0x9d5a000, 0x9, 0x82b9bf4, 0xe, 0x82b739e, 0x1, 0x1, 0x0, ...)
    /home/user/zdns/modules/miekg/miekg.go:545 +0x9a3
github.com/zmap/zdns/modules/miekg.(*Lookup).iterativeLookup(0x9d6e030, 0x10001, 0x9d5a000, 0x9, 0x82b9bf4, 0xe, 0x1, 0x82b739e, 0x1, 0x84aeae4, ...)
    /home/user/zdns/modules/miekg/miekg.go:770 +0x205
github.com/zmap/zdns/modules/miekg.(*Lookup).DoMiekgLookup(0x9d6e030, 0x9d5a000, 0x9, 0x82b9bf4, 0xe, 0x81ba81f, 0x8278280, 0x9c7c6e0, 0x82b742d, 0x2, ...)
    /home/user/zdns/modules/miekg/miekg.go:826 +0x2db
github.com/zmap/zdns/modules/miekg.(*Lookup).DoLookup(0x9d6e030, 0x9d5a000, 0x9, 0x0, 0x0, 0x0, 0x9d5a000, 0x9, 0x0, 0x0, ...)
    /home/user/zdns/modules/miekg/miekg.go:916 +0x45
github.com/zmap/zdns.doLookup(0x83125c0, 0x9c90390, 0x9cb20d0, 0x9c6e5c0, 0x9c6e600, 0x9ce9c80, 0x9c82920, 0x0, 0xa6a52060, 0x5)
    /home/user/zdns/lookup.go:132 +0x7d8
created by github.com/zmap/zdns.DoLookups
    /home/user/zdns/lookup.go:205 +0x287

Thanks!

[dadrian]

@theloglizard What's the output of go version?

[theloglizard]

go version go version go1.11.13 linux/386

[dadrian]

@theloglizard Try building again using a more recent version of Golang (at least 1.13). It doesn't look like your Go version is respecting the replace directive in the go.mod file.

[theloglizard]

OK, I upgraded to go 1.14.4, deleted the previous builds (I think), and the issue remains.

The make clean was pretty minimal, so I deleted my gocode/bin and gocode/pkg which seemed to force the refetch of some dependencies, which seemed a generally good thing. The integration tests were a considerable disaster (see attached for stderr), and much like what's below.

. _ sudo snap refresh --classic --channel=1.14/stable go rm -rf ~/gocode/bin rm -rf ~/gocode/pkg make install (from the source directory with everything)

go build (from the zdns sub directory so I could run the integration tests which expect zdns/zdns)

Checking the one in my path coincides with the date and time I built-- it does. Debian:~/zdns$ ls -l `which zdns` -rwxr-xr-x 1 username username 7103083 Jun 6 08:01 /home/username/gocode/bin/zdns

Debian:\~/zdns$ date Sat 06 Jun 2020 08:07:40 AM

Debian:\~/zdns$ go version go version go1.14.4 linux/386

Debian:\~/zdns$ echo "censys.io" | zdns A {"data":{"answers":[{"answer":"104.26.11.245","class":"IN","name":"censys.io","ttl":300,"type":"A"},{"answer":"104.26.10.245","class":"IN","name":"censys.io","ttl":300,"type":"A"},{"answer":"172.67.68.81","class":"IN","name":"censys.io","ttl":300,"type":"A"}],"protocol":"udp","resolver":"192.168.0.1:53"},"name":"censys.io","status":"NOERROR","timestamp":"2020-06-06T08:07:55-04:00"}

Debian:\~/zdns$ echo "censys.io" | zdns A -iterative panic: runtime error: index out of range [-3624]

goroutine 12 [running]: github.com/zmap/zdns/cachehash.(ShardedCacheHash).getShard(...) /home/username/zdns/cachehash/shardedcachehash.go:42 github.com/zmap/zdns/cachehash.(ShardedCacheHash).Lock(0xa062394, 0x82bcc80, 0xa09c090) /home/username/zdns/cachehash/shardedcachehash.go:72 +0x5b github.com/zmap/zdns/modules/miekg.(GlobalLookupFactory).GetCachedResult(0xa062390, 0xa090007, 0x2, 0x10002, 0x3, 0x4, 0x0, 0x0, 0x0, 0x0, ...) /home/username/zdns/modules/miekg/miekg.go:194 +0x1bc github.com/zmap/zdns/modules/miekg.(Lookup).cachedRetryingLookup(0xa096030, 0x10001, 0xa090000, 0x9, 0x82f9b0f, 0x10, 0x82f671d, 0x1, 0x1, 0x0, ...) /home/username/zdns/modules/miekg/miekg.go:545 +0x825 github.com/zmap/zdns/modules/miekg.(Lookup).iterativeLookup(0xa096030, 0x10001, 0xa090000, 0x9, 0x82f9b0f, 0x10, 0x1, 0x82f671d, 0x1, 0x851b614, ...) /home/username/zdns/modules/miekg/miekg.go:770 +0xd1 github.com/zmap/zdns/modules/miekg.(Lookup).DoMiekgLookup(0xa096030, 0xa090000, 0x9, 0x82f9b0f, 0x10, 0x82031ee, 0x82b4800, 0xa0127e0, 0x82f67a8, 0x2, ...) /home/username/zdns/modules/miekg/miekg.go:826 +0x281 github.com/zmap/zdns/modules/miekg.(*Lookup).DoLookup(0xa096030, 0xa090000, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/username/zdns/modules/miekg/miekg.go:916 +0x45 github.com/zmap/zdns.doLookup(0x8361020, 0xa062390, 0xa10a270, 0xa022800, 0xa022840, 0xa13fd80, 0xa016b90, 0x4, 0x200, 0x28) /home/username/zdns/lookup.go:132 +0x81a created by github.com/zmap/zdns.DoLookups /home/username/zdns/lookup.go:205 +0x2a7_

integration.err.txt

[phillip-stephens]

I'm not entirely sure what was going on here, but I just used the used the oldest version of the Go toolchain we now support (Go 1.20) and in Debian installed zdns from source. I'm able to run both --iterative and external lookups without getting this issue.

I'm going to close this, if someone else runs into it again we can investigate from there.