DNS issue with go 1.19.3?

[jung-kurt]

When gokr-packer is built with go 1.19.3, I get the following error when calling it to update an instance:

2022/11/26 10:11:52 probing url for https: Get "http://jung:1080": dial tcp: lookup jung on 192.168.1.254:53: no such host

This error occurs in internal/httpclient/httpclient.go

probeClient := &http.Client{                                                                                                                                                
  CheckRedirect: func(*http.Request, []*http.Request) error {                                                                                                               
    return http.ErrUseLastResponse // do not follow redirects                                                                                                               
  },                                                                                                                                                                        
}                                                                                                                                                                           
probeResp, err := probeClient.Get("http://" + baseUrl.Host)                                                                                                                 
if err != nil {                                                                                                                                                             
  return "", fmt.Errorf("probing url for https: %v", err)                                                                                                                   
}                      

I think this problem started after I upgraded go from 1.19.2 to 1.19.3. When I build gokr-packer with go 1.18.8 everything works as expected.

Can anyone tell if this a problem with

• the new version of go,
• the way the http.Client structure gets set up,
• or the way my router or environment is set up?

[stapelberg]

Hey, thanks for reporting this issue.

There were changes in Go 1.19 in the DNS resolution regarding EDNS: https://go.dev/doc/go1.19 (paragraph “net”).

Can you check if the issue starts with Go 1.19, or with Go 1.19.3 specifically please?

You could also try reproducing the issue in a standalone minimal program. Try running it with GODEBUG=netdns=cgo or GODEBUG=netdns=go to narrow it down further.

Once you know which version introduces the problem, you can use git bisection (google “git bisection” for tutorials) to figure out the specific git commit between Go 1.18 and Go 1.19 that causes the issue, in just a few steps. (Go compiles really quickly, so this shouldn’t take too long.)

Thanks

[jung-kurt]

Thanks for the quick response with great advice to pinpoint this problem.

For convenience, I have recently used binary Go releases so I have only the most recent point versions of 1.18 and 1.19. I will download the source to zero in on the change in Go's net library that introduced this. I frequently update my gokrazy instance and the problem first occurred with go 1.19.3, so I will focus my attention on the changes from go 1.19.2 to 1.19.3.

In the meantime, running an up-to-date gokr-packer built with go 1.19.3 succeeds when run with the GODEBUG=netdns=cgo environment variable and fails when run with GODEBUG=netdns=go. It was thoughtful of the go authors to provide a workaround for the native go resolver.

[jung-kurt]

Can you check if the issue starts with Go 1.19, or with Go 1.19.3 specifically please?

I followed your plan of action and used git bisection on the go repository and tested it with a short five-liner. Even though the toolchain builds relatively quickly, I wrote a script to be used in conjunction with git bisect run so I would not need to babysit the process. The offending commit (9d34fc5108d91bfcce3c465069dbb2c563af4229) is after version 1.18.8 and before version 1.19. It comprises over 1500 lines of mostly platform-specific assembly code that deals with pipe2().

In retrospect, I must have only recently rebuilt gokr-packer so I was misled into thinking that the problem was introduced with go version 1.19.3 specifically. My previous version of gokr-packer must have been built with a toolchain that preceded the offending commit.

I'll poke this bear a little harder to see if I can find how pipe2() is implicated.

[stapelberg]

Huh, it is surprising to see pipe-related code being pinpointed as the culprit here.

Plus, the change in question is from 2022-03-03, so it’s odd that nobody else reported it yet (AFAICT).

Are you running in a weird environment? Something else than Linux on amd64?

Either way, it would probably be best to take this issue to the Go issue tracker at https://github.com/golang/go/issues

[jung-kurt]

Huh, it is surprising to see pipe-related code being pinpointed as the culprit here.

I agree. Maybe in Go's transition from libc to native Go for DNS, they have a fallback to use an external program in some cases. I'd like to use delve to dig deeper but I don't know how to have it use a specified go toolchain.

Plus, the change in question is from 2022-03-03, so it’s odd that nobody else reported it yet (AFAICT).

I agree again. In my case, I had been using a gokr-packer that had been built before the change.

Are you running in a weird environment? Something else than Linux on amd64?

Old (~2005) but standard:

$ uname -a
Linux sony2 4.19.0-22-amd64 #1 SMP Debian 4.19.260-1 (2022-09-29) x86_64 GNU/Linux

Either way, it would probably be best to take this issue to the Go issue tracker at https://github.com/golang/go/issues

Will do. Thanks for your comments.

Closing.