search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
122.86k stars 17.52k forks source link

net: LookupTXT regards a TXT record with multiple strings as multiple records on Windows #21472

Closed cedar10bits closed 6 years ago

cedar10bits commented 7 years ago

What version of Go are you using (go version)?

go1.7.6 windows/amd64

What did you do?

"example.test.local" is a domain which has a TXT record with multiple strings as below.

"v=spf1 ip4:1.2.3.4 " "~all"

I resolve TXT record of "example.test.local".

txts, err := net.LookupTXT("example.test.local")

What did you expect to see?

On Linux, behave like below(go1.7.6 linux/amd64).

What did you see instead?

On Windows, behave like below.

davecheney commented 7 years ago

Can you please include the output reported by dig.

On 16 Aug 2017, at 21:51, cedar10bits notifications@github.com wrote:

What version of Go are you using (go version)?

go1.7.6 windows/amd64

What did you do?

"example.test.local" is a domain which has a TXT record with multiple strings like below. "v=spf1 ip4:1.2.3.4 " "~all"

I resolve TXT record of "example.test.local". txts, err := net.LookupTXT("example.test.local")

What did you expect to see?

On Linux, behave like below(go1.7.6 linux/amd64).

len(txts) == 1 txts[0] == "v=spf1 ip4:1.2.3.4 ~all" What did you see instead?

On Windows, behave like below.

len(txts) == 2 txts[0] == "v=spf1 ip4:1.2.3.4" txts[1] == "~all" — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

davecheney commented 7 years ago

The relevant rfc for this discussion is, https://tools.ietf.org/html/rfc1464

On 16 Aug 2017, at 21:58, Dave Cheney dave@cheney.net wrote:

Can you please include the output reported by dig.

On 16 Aug 2017, at 21:51, cedar10bits notifications@github.com wrote:

What version of Go are you using (go version)?

go1.7.6 windows/amd64

What did you do?

"example.test.local" is a domain which has a TXT record with multiple strings like below. "v=spf1 ip4:1.2.3.4 " "~all"

I resolve TXT record of "example.test.local". txts, err := net.LookupTXT("example.test.local")

What did you expect to see?

On Linux, behave like below(go1.7.6 linux/amd64).

len(txts) == 1 txts[0] == "v=spf1 ip4:1.2.3.4 ~all" What did you see instead?

On Windows, behave like below.

len(txts) == 2 txts[0] == "v=spf1 ip4:1.2.3.4" txts[1] == "~all" — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

cedar10bits commented 7 years ago

Can you please include the output reported by dig.

# dig example.test.local txt ;; ANSWER SECTION: example.test.local. 0 IN TXT "v=spf1 ip4:1.2.3.4 " "~all"

ianlancetaylor commented 7 years ago

Looks like on Windows DnsQuery_W returns a DNSTXTData record in which the different strings are split up in some way. Perhaps Resolver.lookupTXT in net/lookup_windows.go should be concatenating all the strings in a single TXT record.

cedar10bits commented 7 years ago

This patch works well. lookup_windows.go.patch.txt

ianlancetaylor commented 7 years ago

@cedar10bits Thanks. To avoid any licensing confusion, we will only look at patches that go through the regular contribution process described at https://golang.org/doc/contribute. Would you be willing to send your patch through that?

davecheney commented 7 years ago

I've been researching this this morning and I'm not sure this txt record is well formed. Paging @miekg to issue triage. Stat.

miekg commented 7 years ago

Relevant RFC is 1035, 1464 is EXPERIMENTAL. Text describing the rdata of the txt record:

.  <character-string> is a single
length octet followed by that number of characters.  <character-string>
is treated as binary information, and can be up to 256 characters in
length (including the length octet).

what windows does is completely legal. concatenating the txt segments into one won't work (and I would almost say is illegal), because this will break DNSSEC.

Sorry it took 8 hours, I was asleep

cedar10bits commented 7 years ago

@miekg Thanks. But I think this is not the reason whether what Windows does is legal or not.

. \<character-string> is a single length octet followed by that number of characters. is treated as binary information, and can be up to 256 characters in length (including the length octet).

The above describes only \<character-string> format. TXT record format is described as below in RFC1035.

3.3.14. TXT RDATA format

    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                   TXT-DATA                    /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

where:

TXT-DATA        One or more <character-string>s.

TXT RRs are used to hold descriptive text.  The semantics of the text
depends on the domain where it is found.

what windows does is completely legal. concatenating the txt segments into one won't work (and I would almost say is illegal), because this will break DNSSEC.

And if what Windows does is completely legal, what Linux does will be illegal.

miekg commented 7 years ago

I'm sorry, I'm lost here. This bug is about behavior on windows.

cedar10bits commented 7 years ago

I think this bug is about different behavior between on Windows and on Linux. And I think behavior on Linux is legal.

cedar10bits commented 7 years ago

For your reference, nslookup on Windows returns these results below.

[multiple TXT records]

> nslookup -type=TXT multi.test.local
Server:  UnKnown
Address:  XXX.XXX.XXX.XXX
multi.test.local        text =
        "v=spf1 ip4:2.3.4.5 ~all"
multi.test.local        text =
        "v=spf1 ip4:1.2.3.4 ~all"

[single TXT record with multiple strings]

>nslookup -type=TXT example.test.local
Server:  UnKnown
Address:  192.168.203.175
example.test.local      text =
        "v=spf1 ip4:1.2.3.4 "
        "~all"
alexbrainman commented 7 years ago

I am happy to change code, but I would like to see the problem for myself first. @cedar10bits do you have an example DNS server that I can use to reproduce this problem? Or alternatively maybe someone can suggest a way to easily setup / configure this server on Windows or Linux. Thank you.

Alex

cedar10bits commented 7 years ago

@cedar10bits do you have an example DNS server that I can use to reproduce this problem?

@alexbrainman I don't have my own DNS server which can access globally. But I find reprodusable domains and I list them below. You can reproduce this problem.

alexbrainman commented 7 years ago

But I find reprodusable domains and I list them below. You can reproduce this problem.

I will try when I have some time. Thank you @cedar10bits

Alex

miekg commented 7 years ago

Ah yes, if the zone file has this: "v=spf1 ip4:1.2.3.4 " "~all"

Then what happens on Windows is correct, what happen under Linux is not correct. I.e. you should receive 2 txts elements.

Do you want them to be concatenated when returned from the API? Probably not, because the splitting might be done on purpose (by the zone administrator). Also in your example you've added a space at the end of the first piece (conveniently), if removed: `"v=spf1 ip4:1.2.3.4" "~all" What are you going to do? Insert spaces while concatenating?

alexbrainman commented 7 years ago

I run this program:

package main

import (
    "fmt"
    "net"
    "sort"
)

func main() {
    names := []string{
        "bbc.com",
        "cisco.com",
        "dell.com",
        "ed.gov",
        "nature.com",
        "opera.com",
        "stanford.edu",
    }
    for _, name := range names {
        value, err := net.LookupTXT(name)
        if err != nil {
            panic(err)
        }
        sort.Strings(value)
        fmt.Printf("%q -> %q\n", name, value)
    }
}

on Linux:

"bbc.com" -> ["Fzj91DPhHcxL3FxKMiBraJ9CajRin4nqr8AxflyEQLI+dM+xdOt5/I8F4xGMWelgP2SwFda7w8U2KZFjDR6Ocg==" "MS=ms25863558" "dropbox-domain-verification=mtgv0f2pudoz" "v=spf1 ip4:212.58.224.0/19 ip4:132.185.0.0/16 ip4:78.136.53.80/28 ip4:78.136.14.192/27 ip4:78.136.19.8/29 ip4:89.234.10.72/29 ip4:74.112.66.33 ip4:208.251.80.51 ip4:89.202.185.0/24 ip4:207.159.133.98 ip4:207.159.133.99 include:msgfocus.com include:cmail1.com include:mktomail.com include:servers.mcsv.net include:redsnapper.net ?all"]
"cisco.com" -> ["926723159-3188410" "MS=ms65960035" "docusign=5e18de8e-36d0-4a8e-8e88-b7803423fa2f" "docusign=95052c5f-a421-4594-9227-02ad2d86dfbe" "google-site-verification=K2w--6oeqrFjHfYtTsYyd2tFw7OQd6g5HJDC9UAI8Jk" "v=spf1 ip4:173.37.147.224/27 ip4:173.37.142.64/26 ip4:173.38.212.128/27 ip4:173.38.203.0/24 ip4:64.100.0.0/14 ip4:72.163.7.160/27 ip4:72.163.197.0/24 ip4:144.254.0.0/16 ip4:66.187.208.0/20 ip4:173.37.86.0/24 ip4:64.104.206.0/24 ip4:64.104.15.96/27 ip4:64.102.19.192/26 ip4:144.254.15.96/27 ip4:173.36.137.128/26 ip4:173.36.130.0/24 mx:res.cisco.com mx:sco.cisco.com ~all"]
"dell.com" -> ["google-site-verification=YogqtnoKF39HPLDsBmWJ1MEDE61FlW9mxcv8pj0YnHs" "v=spf1 ip4:143.166.85.0/24 ip4:143.166.148.0/24 ip4:211.130.110.88 ip4:143.166.82.0/24 ip4:143.166.224.0/24 ip4:202.188.162.48/26 ip4:125.252.73.0/24 ip4:211.25.230.0/24 ip4:132.237.17.0/24 ip4:68.232.153.90 ip4:68.232.149.220 ip4:68.232.153.94 ip4:68.232.149.214 ip4:68.232.153.95 ip4:68.232.149.229 ip4:68.232.153.96 ip4:68.232.149.218 ~all" "zpJgV1vrEp/xXNL+QgrMsBQZ4b4teTyu+7GveMS/VpCGC/f4UAUgBVqsp4vsdoOlTW/abanUeQZLcGV6mktRig=="]
"ed.gov" -> ["MS=ms39866724" "v=spf1 include:spf.protection.outlook.com a:crdc2011.appiancloud.com include:sendgrid.net include:spf-a.rnmk.com include:spf-b.rnmk.com include:_spf.salesforce.com ip4:165.224.212.0/23 ip4:165.224.214.0/24 ip4:165.224.20.0/23 ip4:165.224.22.0/23 ip4:165.224.210.0/23 ip4:96.127.34.41 ip6:2610:e8::/32 ip4:12.150.182.154 ip4:204.94.67.74 ip4:63.145.228.10 ip6:2600:1:0:1B:0:0:0:74/32 ip4:165.224.250.0/24 ip4:165.224.249.0/24 ~all" "xbJsf3kVj8DeH8o01d2Iq4SNhOtmJpE7ZjCYDQ0Vrv0rSAvaEuVMhQrbEL2BEm5207Vh5o3ZnxPbepaFAo35QQ=="]
"nature.com" -> ["Hello GlobalSign CEOS1602043687" "google-site-verification=ieCNIjjta99aFWzeD9Mhze-lTbXHZo6GAc-MgXlclL4" "v=spf1 ip4:31.221.90.56/32 ip4:195.128.10.18/32 ip4:31.221.90.43/32 ip4:195.128.10.15/32 ip4:195.128.10.69/32 ip4:203.200.192.105/32 ip4:203.200.192.109/32 ip4:167.89.16.99/32 ip4:31.221.90.34/32 ip4:31.221.90.41/32 ip4:208.85.55.170/32 ip4:208.85.55.173/32 include:madgexjb.com include:_spf.google.com include:spf.mandrillapp.com include:_spf.sparkpostmail.com -all"]
"opera.com" -> ["google-site-verification=zsu8s2znTOAuZ0dkksfMdQE3HmkoNNyuijNib1xkiQo" "v=spf1 ip4:185.26.183.134 ip6:2001:4c28:4000:727:185:26:183:134 ip4:185.26.183.145 ip4:185.26.183.176 ip4:185.26.183.153  ip6:2001:4c28:4000:727:185:26:183:176 ip6:2001:4c28:4000:727:185:26:183:153 ip4:91.203.97.254 ip4:213.236.208.188 ip4:50.97.60.135 ip4:184.173.169.88 ip4:107.167.119.206 ip4:91.123.56.240 ip4:91.123.56.241 ip4:193.75.92.155 ip4:193.75.92.156 mx include:_spf.google.com -all"]
"stanford.edu" -> ["blitz=mu-0e24b17b-23dce65a-c0b14b75-e742cd66" "bwZknJMLNV1XuaJAyUmKlAFrdZo+p5yDlTNACmDUWhgtyihfJc8oWMnK7hWLreN+ozU3mX91yHHzZx0adJPPPg==" "firebase=cartalab" "globalsign-domain-verification=IKP42b_14E6c1gUHTYKgXqj3o5aasL2hd9ECM1tSJP" "htBOpqv5il135Xfa09GRiVHY09Xb9qXNj6o5lAfj8gNoBvyyGu5rSb4gvSj9lCUwY1NF0+wL2BO1ZDIcGIpADQ==" "v=spf1 ip4:171.67.219.64/27 ip4:171.67.224.0/28 ip4:171.67.214.160/27 ip4:171.67.216.240/28 ip4:171.67.43.137 ip4:171.67.43.138 ip4:171.67.43.139 ip4:171.67.43.140 ip4:171.67.43.141 ip4:204.63.229.192/28 include:_spf.google.com include:_spf.qualtrics.com include:icpbounce.com ip4:148.163.149.245 ip4:148.163.153.235 ?all"]

and on Windows:

"bbc.com" -> [" include:msgfocus.com include:cmail1.com include:mktomail.com include:servers.mcsv.net include:redsnapper.net ?all" "Fzj91DPhHcxL3FxKMiBraJ9CajRin4nqr8AxflyEQLI+dM+xdOt5/I8F4xGMWelgP2SwFda7w8U2KZFjDR6Ocg==" "MS=ms25863558" "dropbox-domain-verification=mtgv0f2pudoz" "v=spf1 ip4:212.58.224.0/19 ip4:132.185.0.0/16 ip4:78.136.53.80/28 ip4:78.136.14.192/27 ip4:78.136.19.8/29 ip4:89.234.10.72/29 ip4:74.112.66.33 ip4:208.251.80.51 ip4:89.202.185.0/24 ip4:207.159.133.98 ip4:207.159.133.99"]
"cisco.com" -> [" ip4:64.104.206.0/24 ip4:64.104.15.96/27 ip4:64.102.19.192/26 ip4:144.254.15.96/27 ip4:173.36.137.128/26 ip4:173.36.130.0/24 mx:res.cisco.com mx:sco.cisco.com ~all" "926723159-3188410" "MS=ms65960035" "docusign=5e18de8e-36d0-4a8e-8e88-b7803423fa2f" "docusign=95052c5f-a421-4594-9227-02ad2d86dfbe" "google-site-verification=K2w--6oeqrFjHfYtTsYyd2tFw7OQd6g5HJDC9UAI8Jk" "v=spf1 ip4:173.37.147.224/27 ip4:173.37.142.64/26 ip4:173.38.212.128/27 ip4:173.38.203.0/24 ip4:64.100.0.0/14 ip4:72.163.7.160/27 ip4:72.163.197.0/24 ip4:144.254.0.0/16 ip4:66.187.208.0/20 ip4:173.37.86.0/24"]
"dell.com" -> [".149.214 ip4:68.232.153.95 ip4:68.232.149.229 ip4:68.232.153.96 ip4:68.232.149.218 ~all" "google-site-verification=YogqtnoKF39HPLDsBmWJ1MEDE61FlW9mxcv8pj0YnHs" "v=spf1 ip4:143.166.85.0/24 ip4:143.166.148.0/24 ip4:211.130.110.88 ip4:143.166.82.0/24 ip4:143.166.224.0/24 ip4:202.188.162.48/26 ip4:125.252.73.0/24 ip4:211.25.230.0/24 ip4:132.237.17.0/24 ip4:68.232.153.90 ip4:68.232.149.220 ip4:68.232.153.94 ip4:68.232" "zpJgV1vrEp/xXNL+QgrMsBQZ4b4teTyu+7GveMS/VpCGC/f4UAUgBVqsp4vsdoOlTW/abanUeQZLcGV6mktRig=="]
"ed.gov" -> ["24.210.0/23 ip4:96.127.34.41 ip6:2610:e8::/32 ip4:12.150.182.154 ip4:204.94.67.74 ip4:63.145.228.10 ip6:2600:1:0:1B:0:0:0:74/32 ip4:165.224.250.0/24 ip4:165.224.249.0/24 ~all" "MS=ms39866724" "v=spf1 include:spf.protection.outlook.com a:crdc2011.appiancloud.com include:sendgrid.net include:spf-a.rnmk.com include:spf-b.rnmk.com include:_spf.salesforce.com ip4:165.224.212.0/23 ip4:165.224.214.0/24 ip4:165.224.20.0/23 ip4:165.224.22.0/23 ip4:165.2" "xbJsf3kVj8DeH8o01d2Iq4SNhOtmJpE7ZjCYDQ0Vrv0rSAvaEuVMhQrbEL2BEm5207Vh5o3ZnxPbepaFAo35QQ=="]
"nature.com" -> ["Hello GlobalSign CEOS1602043687" "google-site-verification=ieCNIjjta99aFWzeD9Mhze-lTbXHZo6GAc-MgXlclL4" "ip4:31.221.90.34/32 ip4:31.221.90.41/32 ip4:208.85.55.170/32 ip4:208.85.55.173/32 include:madgexjb.com include:_spf.google.com include:spf.mandrillapp.com include:_spf.sparkpostmail.com -all" "v=spf1 ip4:31.221.90.56/32 ip4:195.128.10.18/32 ip4:31.221.90.43/32 ip4:195.128.10.15/32 ip4:195.128.10.69/32 ip4:203.200.192.105/32 ip4:203.200.192.109/32 ip4:167.89.16.99/32 "]
"opera.com" -> ["4:184.173.169.88 ip4:107.167.119.206 ip4:91.123.56.240 ip4:91.123.56.241 ip4:193.75.92.155 ip4:193.75.92.156 mx include:_spf.google.com -all" "google-site-verification=zsu8s2znTOAuZ0dkksfMdQE3HmkoNNyuijNib1xkiQo" "v=spf1 ip4:185.26.183.134 ip6:2001:4c28:4000:727:185:26:183:134 ip4:185.26.183.145 ip4:185.26.183.176 ip4:185.26.183.153  ip6:2001:4c28:4000:727:185:26:183:176 ip6:2001:4c28:4000:727:185:26:183:153 ip4:91.203.97.254 ip4:213.236.208.188 ip4:50.97.60.135 ip"]
"stanford.edu" -> ["blitz=mu-0e24b17b-23dce65a-c0b14b75-e742cd66" "bwZknJMLNV1XuaJAyUmKlAFrdZo+p5yDlTNACmDUWhgtyihfJc8oWMnK7hWLreN+ozU3mX91yHHzZx0adJPPPg==" "firebase=cartalab" "globalsign-domain-verification=IKP42b_14E6c1gUHTYKgXqj3o5aasL2hd9ECM1tSJP" "htBOpqv5il135Xfa09GRiVHY09Xb9qXNj6o5lAfj8gNoBvyyGu5rSb4gvSj9lCUwY1NF0+wL2BO1ZDIcGIpADQ==" "include:_spf.google.com include:_spf.qualtrics.com include:icpbounce.com ip4:148.163.149.245 ip4:148.163.153.235 ?all" "v=spf1 ip4:171.67.219.64/27 ip4:171.67.224.0/28 ip4:171.67.214.160/27 ip4:171.67.216.240/28 ip4:171.67.43.137 ip4:171.67.43.138 ip4:171.67.43.139 ip4:171.67.43.140 ip4:171.67.43.141 ip4:204.63.229.192/28 "]

@miekg which version needs to be fixed? I suspect Windows is wrong here. Thank you.

Alex

cedar10bits commented 7 years ago

Do you want them to be concatenated when returned from the API? Probably not, because the splitting might be done on purpose (by the zone administrator).

@miekg I think so, too. I think this API should be defined as below, ideally.

type TXTRecord struct {
    Txt []string
}
func LookupTXT(name string) ([]TXTRecord, error)

But this API is defined as a function that returns multiple TXT records of a string array (https://golang.org/pkg/net/#LookupTXT). @alexbrainman shows difference between Windows and Linux. On Windows, a record with multiple strings is treated as multiple records with single string. I think this behavior is not followed by the definition of this API.

cedar10bits commented 7 years ago

Also in your example you've added a space at the end of the first piece (conveniently), if removed: `"v=spf1 ip4:1.2.3.4" "~all" What are you going to do? Insert spaces while concatenating?

@miekg If removed, just concatenate them without adding spaces. In this case, I expect "v=spf1 ip4:1.2.3.4\~all" (no space between "1.2.3.4" and "\~all"). It is useful for some cases, for example, SPF (https://tools.ietf.org/html/rfc7208).

3.3.  Multiple Strings in a Single DNS Record

   As defined in [RFC1035], Sections 3.3 and 3.3.14, a single text DNS
   record can be composed of more than one string.  If a published
   record contains multiple character-strings, then the record MUST be
   treated as if those strings are concatenated together without adding
   spaces.  For example:

      IN TXT "v=spf1 .... first" "second string..."

   is equivalent to:

      IN TXT "v=spf1 .... firstsecond string..."
miekg commented 7 years ago

But LookupTXT has no idea it deals with SPF records (oh if only they had used a new record type...), so following those rules in 7208 doesn't work for all cases.

@alexbrainman using

package main

import (
    "fmt"
    "net"
    "sort"
)

func main() {
    names := []string{
        "bbc.com",
        "dell.com",
    }
    for _, name := range names {
        value, err := net.LookupTXT(name)
        if err != nil {
            panic(err)
        }
        sort.Strings(value)

        fmt.Println(name)
        for _, v := range value {
            fmt.Printf("  * %q\n", v)
        }
    }
}

Building this with CGO_ENABLED=0 go build lookup.go as we don't want to involve glibc here (assuming) And checking what this gives (on linux) does have:

bbc.com
  * "Fzj91DPhHcxL3FxKMiBraJ9CajRin4nqr8AxflyEQLI+dM+xdOt5/I8F4xGMWelgP2SwFda7w8U2KZFjDR6Ocg=="
  * "MS=ms25863558"
  * "dropbox-domain-verification=mtgv0f2pudoz"
  * "v=spf1 ip4:212.58.224.0/19 ip4:132.185.0.0/16 ip4:78.136.53.80/28 ip4:78.136.14.192/27 ip4:78.136.19.8/29 ip4:89.234.10.72/29 ip4:74.112.66.33 ip4:208.251.80.51 ip4:89.202.185.0/24 ip4:207.159.133.98 ip4:207.159.133.99 include:msgfocus.com include:cmail1.com include:mktomail.com include:servers.mcsv.net include:redsnapper.net ?all"

Checking with dig:

;bbc.com.           IN  TXT

;; ANSWER SECTION:
bbc.com.        497 IN  TXT "Fzj91DPhHcxL3FxKMiBraJ9CajRin4nqr8AxflyEQLI+dM+xdOt5/I8F4xGMWelgP2SwFda7w8U2KZFjDR6Ocg=="
bbc.com.        497 IN  TXT "MS=ms25863558"
bbc.com.        497 IN  TXT "dropbox-domain-verification=mtgv0f2pudoz"
bbc.com.        497 IN  TXT "v=spf1 ip4:212.58.224.0/19 ip4:132.185.0.0/16 ip4:78.136.53.80/28 ip4:78.136.14.192/27 ip4:78.136.19.8/29 ip4:89.234.10.72/29 ip4:74.112.66.33 ip4:208.251.80.51 ip4:89.202.185.0/24 ip4:207.159.133.98 ip4:207.159.133.99" " include:msgfocus.com include:cmail1.com include:mktomail.com include:servers.mcsv.net include:redsnapper.net ?all"

So indeed on Linux this look OK (Don't own Windows - so can't check there).

What's wrong with Windows? The order of the different txt segments? (At least that is what I can make out). There is no inherent order of these TXT records, the can be shuffled around as you please. If you want to insists there is only 1 ordering: the canonical DNS ordering (RFC 4034).

It could be that on the windows tests the TXT order is different (unlikely). What does dig on windows show? Is Go doing some sorting (on Windows)?

cedar10bits commented 7 years ago

But LookupTXT has no idea it deals with SPF records (oh if only they had used a new record type...), so following those rules in 7208 doesn't work for all cases.

I know those rules are only for SPF (multiple strings), not for TXT records. That's OK.

Here is what nslookup gives on Windows (instead of dig).

>nslookup -type=TXT bbc.com
Server:  UnKnown
Address:  XXX.XXX.XXX.XXX

Non-authoritative answer:
bbc.com text =

        "MS=ms25863558"
bbc.com text =

        "dropbox-domain-verification=mtgv0f2pudoz"
bbc.com text =

        "Fzj91DPhHcxL3FxKMiBraJ9CajRin4nqr8AxflyEQLI+dM+xdOt5/I8F4xGMWelgP2SwFda7w8U2KZFjDR6Ocg=="
bbc.com text =

        "v=spf1 ip4:212.58.224.0/19 ip4:132.185.0.0/16 ip4:78.136.53.80/28 ip4:78.136.14.192/27 ip4:78.136.19.8/29 ip4:89.234.10.72/29 ip4:74.112.66.33 ip4:208.251.80.51 ip4:89.202.185.0/24 ip4:207.159.133.98 ip4:207.159.133.99"
        " include:msgfocus.com include:cmail1.com include:mktomail.com include:servers.mcsv.net include:redsnapper.net ?all"

@alexbrainman using

package main

import (
    "fmt"
    "net"
    // "sort"
)

func main() {
    names := []string{
        "bbc.com",
        "dell.com",
    }
    for _, name := range names {
        value, err := net.LookupTXT(name)
        if err != nil {
            panic(err)
        }
        // sort.Strings(value)

        fmt.Println(name)
        for _, v := range value {
            fmt.Printf("  * %q\n", v)
        }
    }
}

What this gives on Linux (CGO_ENABLED=0) is below:

bbc.com
  * "MS=ms25863558"
  * "dropbox-domain-verification=mtgv0f2pudoz"
  * "Fzj91DPhHcxL3FxKMiBraJ9CajRin4nqr8AxflyEQLI+dM+xdOt5/I8F4xGMWelgP2SwFda7w8U2KZFjDR6Ocg=="
  * "v=spf1 ip4:212.58.224.0/19 ip4:132.185.0.0/16 ip4:78.136.53.80/28 ip4:78.136.14.192/27 ip4:78.136.19.8/29 ip4:89.234.10.72/29 ip4:74.112.66.33 ip4:208.251.80.51 ip4:89.202.185.0/24 ip4:207.159.133.98 ip4:207.159.133.99 include:msgfocus.com include:cmail1.com include:mktomail.com include:servers.mcsv.net include:redsnapper.net ?all"

And what this gives on Windows is below:

bbc.com
  * "Fzj91DPhHcxL3FxKMiBraJ9CajRin4nqr8AxflyEQLI+dM+xdOt5/I8F4xGMWelgP2SwFda7w8U2KZFjDR6Ocg=="
  * "v=spf1 ip4:212.58.224.0/19 ip4:132.185.0.0/16 ip4:78.136.53.80/28 ip4:78.136.14.192/27 ip4:78.136.19.8/29 ip4:89.234.10.72/29 ip4:74.112.66.33 ip4:208.251.80.51 ip4:89.202.185.0/24 ip4:207.159.133.98 ip4:207.159.133.99"
  * " include:msgfocus.com include:cmail1.com include:mktomail.com include:servers.mcsv.net include:redsnapper.net ?all"
  * "MS=ms25863558"
  * "dropbox-domain-verification=mtgv0f2pudoz"

On Windows, "v=spf1 ... " and " include: ... " are regarded as two other TXT records (not single TXT record). Order of TXT records is not stable (on both OS), but " include: ... " always follows "v=spf1 ... " on Windows.

miekg commented 7 years ago

ah thanks @cedar10bits now I see (these strings are too long). On linux the returned segment has a length > 255 (which can't possibly be right if you follow the letter of the RFC). In windows the 2 segments of that TXT record v=spf1 are correctly put into two text elements.

Again Windows code seems correct to me, what happens on Linux is somewhat fishy.

cedar10bits commented 7 years ago

This API is defined as below (https://golang.org/pkg/net/#LookupTXT).

LookupTXT returns the DNS TXT records for the given domain name.

This API returns TXT records as a string array. I think that one element of this string array must be single whole TXT record, not a part of TXT record. v=spf1 is a part of TXT record, but not single whole TXT record. If Windows code is correct, behavior on Windows is not followed by this API's definition.

miekg commented 7 years ago

Well that's DNS for you. What Windows does here makes sense from a DNS perspective.

On Aug 23, 2017 12:25 PM, "cedar10bits" notifications@github.com wrote:

This API is defined as below (https://golang.org/pkg/net/#LookupTXT).

LookupTXT returns the DNS TXT records for the given domain name.

This API returns TXT records as a string array. I think that one element of this string array must be single TXT record, not a part of a TXT record. v=spf1 is a part of a TXT record, but not single TXT record. If Windows code is correct, behavior on Windows is not followed by this API's definition.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/21472#issuecomment-324300023, or mute the thread https://github.com/notifications/unsubscribe-auth/AAVkW2gyqidboyazexrKPmf9k87taMypks5sbAw0gaJpZM4O4zEm .

cedar10bits commented 7 years ago

If what Windows does is correct, I will have no idea to distinguish between single whole TXT record and a part of TXT record. To use this API correctly, I need to distinguish. @miekg do you have any idea ?

miekg commented 7 years ago

I think the answer should be, if you care about that maybe you shouldn't use txt records to store stuff your interested in.

Otoh is seems the API should have returned a slice of slices. Obviously we can't change that anymore. Concatenating txt segments like this might be ok for spf, but may not work for other txt uses. spf is a broken standard seen from the DNS standpoint.

On Aug 23, 2017 12:54 PM, "cedar10bits" notifications@github.com wrote:

If what Windows does is correct, I will have no idea to distinguish between single whole TXT record and a part of TXT record. To use this API correctly, I need to distinguish. @miekg https://github.com/miekg do you have any idea ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/21472#issuecomment-324305677, or mute the thread https://github.com/notifications/unsubscribe-auth/AAVkW71JCnP9Kjr7YHyzicy9B6RthZWDks5sbBL5gaJpZM4O4zEm .

cedar10bits commented 7 years ago

Otoh is seems the API should have returned a slice of slices. Obviously we can't change that anymore.

I think so, too.

Concatenating txt segments like this might be ok for spf, but may not work for other txt uses. spf is a broken standard seen from the DNS standpoint.

I want to know the other txt uses which concatenating may not work for. DNSSEC?

miekg commented 7 years ago

On Aug 23, 2017 1:19 PM, "cedar10bits" notifications@github.com wrote:

Otoh is seems the API should have returned a slice of slices. Obviously we can't change that anymore.

I think so, too.

Concatenating txt segments like this might be ok for spf, but may not work for other txt uses. spf is a broken standard seen from the DNS standpoint.

I want to know the other txt uses which concatenating may not work for. DNSSEC

It changes what the zone administrator originally put in the zone.

If you want to reconstruct the original record to validate the signature, you can't. However in the case of net.LookupTXT that is probably not a big issue.

cedar10bits commented 7 years ago

It changes what the zone administrator originally put in the zone.

I think both behavior on Windows and Linux change what the zone administrator originally put in the zone. @miekg what do you think about it ?

miekg commented 7 years ago

I thought the windows version did it right, or did you see things standing out there as well?

On Aug 23, 2017 2:19 PM, "cedar10bits" notifications@github.com wrote:

It changes what the zone administrator originally put in the zone.

I think both behavior on Windows and Linux change what the zone administrator originally put in the zone. @miekg https://github.com/miekg what do you think about it ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/21472#issuecomment-324327632, or mute the thread https://github.com/notifications/unsubscribe-auth/AAVkW8RfyAP6aOFILnyi5cO4Fu2GLA6Hks5sbCbdgaJpZM4O4zEm .

cedar10bits commented 7 years ago

To understand DNS standpoint, I study DNS zone file. I want to check whether my understandings is correct or not.

Zone file of bbc.com may be written like below (TXT record only):

@       IN  TXT     "MS=ms25863558"
@       IN  TXT     "dropbox-domain-verification=mtgv0f2pudoz"
@       IN  TXT     "Fzj91DPhHcxL3FxKMiBraJ9CajRin4nqr8AxflyEQLI+dM+xdOt5/I8F4xGMWelgP2SwFda7w8U2KZFjDR6Ocg=="
@       IN  TXT     (   "v=spf1 ip4:212.58.224.0/19 ip4:132.185.0.0/16 ip4:78.136.53.80/28 ip4:78.136.14.192/27 ip4:78.136.19.8/29 ip4:89.234.10.72/29 ip4:74.112.66.33 ip4:208.251.80.51 ip4:89.202.185.0/24 ip4:207.159.133.98 ip4:207.159.133.99"
                        " include:msgfocus.com include:cmail1.com include:mktomail.com include:servers.mcsv.net include:redsnapper.net ?all")

On Windows, this API regards zone file as below:

@       IN  TXT     "MS=ms25863558"
@       IN  TXT     "dropbox-domain-verification=mtgv0f2pudoz"
@       IN  TXT     "Fzj91DPhHcxL3FxKMiBraJ9CajRin4nqr8AxflyEQLI+dM+xdOt5/I8F4xGMWelgP2SwFda7w8U2KZFjDR6Ocg=="
@       IN  TXT     "v=spf1 ip4:212.58.224.0/19 ip4:132.185.0.0/16 ip4:78.136.53.80/28 ip4:78.136.14.192/27 ip4:78.136.19.8/29 ip4:89.234.10.72/29 ip4:74.112.66.33 ip4:208.251.80.51 ip4:89.202.185.0/24 ip4:207.159.133.98 ip4:207.159.133.99"
@       IN  TXT     " include:msgfocus.com include:cmail1.com include:mktomail.com include:servers.mcsv.net include:redsnapper.net ?all"

And on Linux, this API regards as below (syntax error occurs on BIND because of too long strings):

@       IN  TXT     "MS=ms25863558"
@       IN  TXT     "dropbox-domain-verification=mtgv0f2pudoz"
@       IN  TXT     "Fzj91DPhHcxL3FxKMiBraJ9CajRin4nqr8AxflyEQLI+dM+xdOt5/I8F4xGMWelgP2SwFda7w8U2KZFjDR6Ocg=="
@       IN  TXT     "v=spf1 ip4:212.58.224.0/19 ip4:132.185.0.0/16 ip4:78.136.53.80/28 ip4:78.136.14.192/27 ip4:78.136.19.8/29 ip4:89.234.10.72/29 ip4:74.112.66.33 ip4:208.251.80.51 ip4:89.202.185.0/24 ip4:207.159.133.98 ip4:207.159.133.99 include:msgfocus.com include:cmail1.com include:mktomail.com include:servers.mcsv.net include:redsnapper.net ?all"

Is the above correct ? If correct, both versions change what the zone administrator originally put in the zone.

miekg commented 7 years ago

I've given my answers in previous comments

alexbrainman commented 7 years ago

I do not have any suggestions. But I decided to implement LookupTXT that returns [][]string instead of []string on windows.

diff --git a/src/net/lookup_windows.go b/src/net/lookup_windows.go
index 0036d89..6994985 100644
--- a/src/net/lookup_windows.go
+++ b/src/net/lookup_windows.go
@@ -287,6 +287,29 @@ func (*Resolver) lookupTXT(ctx context.Context, name string) ([]string, error) {
    return txts, nil
 }

+func WindowsLookupTXT(name string) ([][]string, error) {
+   acquireThread()
+   defer releaseThread()
+   var r *syscall.DNSRecord
+   e := syscall.DnsQuery(name, syscall.DNS_TYPE_TEXT, 0, nil, &r, nil)
+   if e != nil {
+       return nil, &DNSError{Err: winError("dnsquery", e).Error(), Name: name}
+   }
+   defer syscall.DnsRecordListFree(r, 1)
+
+   txts := make([][]string, 0, 10)
+   for _, p := range validRecs(r, syscall.DNS_TYPE_TEXT, name) {
+       d := (*syscall.DNSTXTData)(unsafe.Pointer(&p.Data[0]))
+       txt := make([]string, 0)
+       for _, v := range (*[1 << 10]*uint16)(unsafe.Pointer(&(d.StringArray[0])))[:d.StringCount] {
+           s := syscall.UTF16ToString((*[1 << 20]uint16)(unsafe.Pointer(v))[:])
+           txt = append(txt, s)
+       }
+       txts = append(txts, txt)
+   }
+   return txts, nil
+}
+
 func (*Resolver) lookupAddr(ctx context.Context, addr string) ([]string, error) {
    // TODO(bradfitz): finish ctx plumbing. Nothing currently depends on this.
    acquireThread()

Then this program

package main

import (
    "fmt"
    "net"
)

func main() {
    names := []string{
        "bbc.com",
    }
    for _, name := range names {
        value, err := net.WindowsLookupTXT(name)
        if err != nil {
            panic(err)
        }

        fmt.Println(name)
        for _, v := range value {
            fmt.Printf("  * %q\n", v)
        }
    }
}

outputs

bbc.com
  * ["MS=ms25863558"]
  * ["dropbox-domain-verification=mtgv0f2pudoz"]
  * ["v=spf1 ip4:212.58.224.0/19 ip4:132.185.0.0/16 ip4:78.136.53.80/28 ip4:78.136.14.192/27 ip4:78.136.19.8/29 ip4:89.234.10.72/29 ip4:74.112.66.33 ip4:208.251.80.51 ip4:89.202.185.0/24 ip4:207.159.133.98 ip4:207.159.133.99" " include:msgfocus.com include:cmail1.com include:mktomail.com include:servers.mcsv.net include:redsnapper.net ?all"]
  * ["Fzj91DPhHcxL3FxKMiBraJ9CajRin4nqr8AxflyEQLI+dM+xdOt5/I8F4xGMWelgP2SwFda7w8U2KZFjDR6Ocg=="]

So that would have been perfect. No?

Alex

cedar10bits commented 7 years ago

@miekg I don't think behavior on Windows is legal on DNS standpoint. Because single TXT record may be regarded as multiple TXT records. Isn't it a problem on DNS standpoint?

@alexbrainman Thank you. Maybe I can use it. I wish net.LookupTXT were like this.

alexbrainman commented 7 years ago

I wish net.LookupTXT were like this.

If net.LookupTXT is important enough from security point of view or something, maybe we could mark net.LookupTXT as depreciated and add new net.ZZZLookupTXT function. I do not know enough about this to decide.

Alex

miekg commented 7 years ago

[ Quoting notifications@github.com in "Re: [golang/go] net: LookupTXT rega..." ]

I wish net.LookupTXT were like this.

If net.LookupTXT is important enough from security point of view or something, maybe we could mark net.LookupTXT as depreciated and add new net.ZZZLookupTXT function. I do not know enough about this to decide.

I would consider LookupTXT good enough at this point. If you need more, you'll need to use a specialized DNS package (which requires more code, but gives you far more control - as so far DNS lets you).

It may be worth adding another TXT lookup function, but not sure what that adds, not everything needs to be in the std lib.

alexbrainman commented 7 years ago

I would consider LookupTXT good enough at this point.

Sounds good to me. @cedar10bits should we close this as unfortunate?

Alex

cedar10bits commented 7 years ago

Sounds good to me. @cedar10bits should we close this as unfortunate?

Sorry, I do not close this.

Package net is portable standard library (see overview). https://golang.org/pkg/net/

net.LookupTXT is not portable because of different behavior between on Windows and on Linux.

alexbrainman commented 7 years ago

Closing as unfortunate.

cedar10bits commented 7 years ago

Why close ? To support legacy code portability, net.LookupTXT should be fixed.

alexbrainman commented 7 years ago

To support legacy code portability, net.LookupTXT should be fixed.

What do you propose we do?

Alex

cedar10bits commented 7 years ago

What do you propose we do?

If we leave this API what it is, this API is not portable. Some codes already use this API and these must be portable. To avoid this, we have two options.

  1. Fix this API to equalize both behavior on Windows and on Linux.
  2. Add new API instead of this API in the future (deprecate this API).

I would like to propose first one (and fix Windows version). But now, option 2 seems to be adopted. If adopt 2, we need to decide when to add and announce it to users.

miekg commented 7 years ago

[ Quoting notifications@github.com in "Re: [golang/go] net: LookupTXT rega..." ]

What do you propose we do?

If we leave this API what it is, this API is not portable. Some codes already use this API and these must be portable. To avoid this, we have two options.

  1. Fix this API to equalize both behavior on Windows and on Linux.
  2. Add new API instead of this API in the future (deprecate this API).

I would like to propose first one (this API should not separate single TXT record). But now, option 2 seems to be adopted. If adopt 2, we need to decide when to add.

Other than the difference between Windows and Linux, what is the actually problem here?

cedar10bits commented 7 years ago

Other than the difference between Windows and Linux, what is the actually problem here?

Almost nothing. If fix the behavior, we cannot decide how to fix. Because both behavior on Windows and Linux is not legal on DNS standpoint. (I think behavior on Linux is better for API users)

alexbrainman commented 7 years ago

@cedar10bits

Fix this API to equalize both behavior on Windows and on Linux.

Since I know nothing about DNS, I won't be able to implement this. So if you want this implemented you have to do it yourself. And I am not sure your change will be accepted - your change might be considered as "changing behavior too much". But you have to try to find out.

Add new API instead of this API in the future (deprecate this API).

I called this option, but I don't see this accepted, unless we can demonstrate that current behavior is very bad (security wise or something). I don't think "the difference between Windows and Linux" is good enough reason to introduce new API.

Alex

cedar10bits commented 6 years ago

@alexbrainman Sorry for replying so late, I had got into some trouble in private.

So if you want this implemented you have to do it yourself.

I will implement this if accepted.

I don't think "the difference between Windows and Linux" is good enough reason to introduce new API.

Having thought about this, I think so, too. So I won't deprecate this API and I want change this API like below.

  1. Change behavior of this API on Windows
  2. Add func LookupTXTWithSep(name, sep string) ([]string, error) (TBD: function name) This function returns multiple TXT records as a string array. Multiple strings in single TXT record are concatenated and the separator string sep is placed between the strings. To use this function, the user can get what the zone administrator originally put in the zone. For example, LookupTXTWithSep("bbc.com", "\n") returns TXT records as below: 
    {
"MS=ms25863558",
"dropbox-domain-verification=mtgv0f2pudoz",
"Fzj91DPhHcxL3FxKMiBraJ9CajRin4nqr8AxflyEQLI+dM+xdOt5/I8F4xGMWelgP2SwFda7w8U2KZFjDR6Ocg==",
"v=spf1 ip4:212.58.224.0/19 ip4:132.185.0.0/16 ip4:78.136.53.80/28 ip4:78.136.14.192/27 ip4:78.136.19.8/29 ip4:89.234.10.72/29 ip4:74.112.66.33 ip4:208.251.80.51 ip4:89.202.185.0/24 ip4:207.159.133.98 ip4:207.159.133.99\n include:msgfocus.com include:cmail1.com include:mktomail.com include:servers.mcsv.net include:redsnapper.net ?all"
// "\n" is placed after "ip4:207.159.133.99".
}

    FYI, LookupTXT(name) is the same as LookupTXTWithSep(name, "").

alexbrainman commented 6 years ago

So I won't deprecate this API and I want change this API like below.

We cannot change existing API because existing programs that use net.LookupTXT will fail to build, so this change is not allowed as per https://golang.org/doc/go1compat

Alex

cedar10bits commented 6 years ago

So I won't deprecate this API and I want change this API like below.

It was said in an inappropriate way. I want to change behavior of LookupTXT(name string) on Windows (concatenating all the strings in a single TXT record). And I want to add extra API, LookupTXTWithSep(name, sep string) to get single TXT record with multiple strings properly. I think these ideas will solve discussed problems.

alexbrainman commented 6 years ago

I want to change behavior of LookupTXT(name string) And I want to add extra API, LookupTXTWithSep(name, sep string)

I am not the person who decides here. If you are OK with your changes to be rejected, feel free to try and send your changes for review.

Alex

miekg commented 6 years ago

[ Quoting notifications@github.com in "Re: [golang/go] net: LookupTXT rega..." ]

I want to change behavior of LookupTXT(name string) And I want to add extra API, LookupTXTWithSep(name, sep string)

I am not the person who decides here. If you are OK with your changes to be rejected, feel free to try and send your changes for review.

  1. was discussed at length in this thread, so I'm not sure why it is being brought back.
  2. Is not a good idea. I'm also entirely unsure why this needs to be in the std lib.

/Miek

-- Miek Gieben