search

turbot / steampipe-plugin-net

Use SQL to instantly query DNS records, certificates and other network information. Open source CLI. No DB required.
https://hub.steampipe.io/plugins/turbot/net
Apache License 2.0
22 stars 5 forks source link

Unable to return TXT records #56

Closed massyn closed 1 year ago

massyn commented 1 year ago

Describe the bug When querying TXT records, some domains do not return any data.

Steampipe version (steampipe -v) steampipe version 0.18.1

Plugin version (steampipe plugin list)

Installed Plugin Version Connections
hub.steampipe.io/plugins/turbot/aws@latest 0.92.1 aws
hub.steampipe.io/plugins/turbot/crowdstrike@latest 0.1.0 crowdstrike
hub.steampipe.io/plugins/turbot/csv@latest 0.5.0
hub.steampipe.io/plugins/turbot/jira@latest 0.8.0 jira
hub.steampipe.io/plugins/turbot/net@latest 0.8.1 net
hub.steampipe.io/plugins/turbot/okta@latest 0.8.0 okta
hub.steampipe.io/plugins/turbot/salesforce@latest 0.2.0 salesforce

To reproduce Steps to reproduce the behavior (please include relevant code and/or commands).

Run the following query in Steampipe

select
  value,
  ttl
from
  net_dns_record
where
  domain = 'github.com'
  and type = 'TXT' and dns_server = '8.8.8.8';

Expected behavior Expecting the result to be similar to nslookup -query=txt github.com 8.8.8.8, however no data is being returned.

Additional context While running the same query on my private domain massyn.net, the result is being returned as expected. It would appear that there may be some characters or text in the output result of the DNS query that is causing the plugin to drop it.

misraved commented 1 year ago

Thanks for raising this issue @massyn 👍.

Relevant slack thread - https://steampipe.slack.com/archives/C01UECB59A7/p1674631153271819

massyn commented 1 year ago

Hi @misraved - what is the expected ETA on having this issue resolved? Just curious as to how long I need to be operating my workaround script while steampipe has this bug.

misraved commented 1 year ago

I'm sorry for the delayed response @massyn, we have successfully managed to reproduce the issue, but the details leading up to it is a bit sparse.

For instance, @Subhajit97 is able to successfully query the TXT records with one of his internet providers however I am facing a similar issue as you are. When he switched to a different internet provider, however, he could not query the TXT records for google.com and github.com.

We managed to query the TXT records for steampipe.io and facebook.com successfully but the results don't seem promising for google.com and github.com. We are looking to nail down the root cause of this issue, and hopefully, towards the end of this week, we will have a solution in place.

Thank you once again for your patience and for providing such detailed insight into the issue.

cbruno10 commented 1 year ago

@massyn I believe the incomplete results are due to truncated results, as the response for GitHub's DNS records is greater than 512 bytes. This results in some records being returned when not specifically looking for TXT records:

> select
  *
from
  net_dns_record
where
  domain = 'github.com' and
  dns_server = '8.8.8.8';
+------------+------+------------+--------------+--------------------------+----------+-----------+----------------+------+------------+---------+---------+--------+---------+---------------------------+
| domain     | type | dns_server | ip           | target                   | priority | tag       | value          | ttl  | serial     | minimum | refresh | retry  | expire  | _ctx                      |
+------------+------+------------+--------------+--------------------------+----------+-----------+----------------+------+------------+---------+---------+--------+---------+---------------------------+
| github.com | A    | 8.8.8.8    | 140.82.112.4 | <null>                   | <null>   | <null>    | <null>         | 60   | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | CAA  | 8.8.8.8    | <null>       | <null>                   | <null>   | issue     | digicert.com   | 3590 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | CAA  | 8.8.8.8    | <null>       | <null>                   | <null>   | issuewild | digicert.com   | 3590 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | CAA  | 8.8.8.8    | <null>       | <null>                   | <null>   | issue     | globalsign.com | 3590 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | MX   | 8.8.8.8    | <null>       | alt2.aspmx.l.google.com. | 5        | <null>    | <null>         | 2245 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | MX   | 8.8.8.8    | <null>       | alt3.aspmx.l.google.com. | 10       | <null>    | <null>         | 2245 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | MX   | 8.8.8.8    | <null>       | alt4.aspmx.l.google.com. | 10       | <null>    | <null>         | 2245 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | MX   | 8.8.8.8    | <null>       | aspmx.l.google.com.      | 1        | <null>    | <null>         | 2245 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | MX   | 8.8.8.8    | <null>       | alt1.aspmx.l.google.com. | 5        | <null>    | <null>         | 2245 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | NS   | 8.8.8.8    | <null>       | ns-520.awsdns-01.net.    | <null>   | <null>    | <null>         | 2306 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | NS   | 8.8.8.8    | <null>       | dns4.p08.nsone.net.      | <null>   | <null>    | <null>         | 2306 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | NS   | 8.8.8.8    | <null>       | ns-1283.awsdns-32.org.   | <null>   | <null>    | <null>         | 2306 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | NS   | 8.8.8.8    | <null>       | dns1.p08.nsone.net.      | <null>   | <null>    | <null>         | 2306 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | NS   | 8.8.8.8    | <null>       | ns-1707.awsdns-21.co.uk. | <null>   | <null>    | <null>         | 2306 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | NS   | 8.8.8.8    | <null>       | dns2.p08.nsone.net.      | <null>   | <null>    | <null>         | 2306 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | NS   | 8.8.8.8    | <null>       | ns-421.awsdns-52.com.    | <null>   | <null>    | <null>         | 2306 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | NS   | 8.8.8.8    | <null>       | dns3.p08.nsone.net.      | <null>   | <null>    | <null>         | 2306 | <null>     | <null>  | <null>  | <null> | <null>  | {"connection_name":"net"} |
| github.com | SOA  | 8.8.8.8    | <null>       | dns1.p08.nsone.net.      | <null>   | <null>    | <null>         | 3236 | 1656468023 | 3600    | 43200   | 7200   | 1209600 | {"connection_name":"net"} |
+------------+------+------------+--------------+--------------------------+----------+-----------+----------------+------+------------+---------+---------+--------+---------+---------------------------+

As an experiment, I increased the buffer size to 4096 with EDNS in https://github.com/turbot/steampipe-plugin-net/commit/869fe3e5027047d50b7e74342c4cfd52365f7b72, which then allowed me to retrieve all of the records (output is truncated for the value column):

> select
  *
from
  net_dns_record
where
  domain = 'github.com' and
  dns_server = '8.8.8.8';
+------------+------+------------+--------------+--------------------------+----------+-----------+-----------------------------------------------------------------------------------------------------------------
| domain     | type | dns_server | ip           | target                   | priority | tag       | value
+------------+------+------------+--------------+--------------------------+----------+-----------+-----------------------------------------------------------------------------------------------------------------
| github.com | A    | 8.8.8.8    | 140.82.114.4 | <null>                   | <null>   | <null>    | <null>
| github.com | CAA  | 8.8.8.8    | <null>       | <null>                   | <null>   | issuewild | digicert.com
| github.com | CAA  | 8.8.8.8    | <null>       | <null>                   | <null>   | issue     | digicert.com
| github.com | CAA  | 8.8.8.8    | <null>       | <null>                   | <null>   | issue     | globalsign.com
| github.com | MX   | 8.8.8.8    | <null>       | alt2.aspmx.l.google.com. | 5        | <null>    | <null>
| github.com | MX   | 8.8.8.8    | <null>       | alt3.aspmx.l.google.com. | 10       | <null>    | <null>
| github.com | MX   | 8.8.8.8    | <null>       | alt4.aspmx.l.google.com. | 10       | <null>    | <null>
| github.com | MX   | 8.8.8.8    | <null>       | alt1.aspmx.l.google.com. | 5        | <null>    | <null>
| github.com | MX   | 8.8.8.8    | <null>       | aspmx.l.google.com.      | 1        | <null>    | <null>
| github.com | NS   | 8.8.8.8    | <null>       | ns-520.awsdns-01.net.    | <null>   | <null>    | <null>
| github.com | NS   | 8.8.8.8    | <null>       | dns4.p08.nsone.net.      | <null>   | <null>    | <null>
| github.com | NS   | 8.8.8.8    | <null>       | dns1.p08.nsone.net.      | <null>   | <null>    | <null>
| github.com | NS   | 8.8.8.8    | <null>       | ns-1283.awsdns-32.org.   | <null>   | <null>    | <null>
| github.com | NS   | 8.8.8.8    | <null>       | dns2.p08.nsone.net.      | <null>   | <null>    | <null>
| github.com | NS   | 8.8.8.8    | <null>       | ns-1707.awsdns-21.co.uk. | <null>   | <null>    | <null>
| github.com | NS   | 8.8.8.8    | <null>       | dns3.p08.nsone.net.      | <null>   | <null>    | <null>
| github.com | NS   | 8.8.8.8    | <null>       | ns-421.awsdns-52.com.    | <null>   | <null>    | <null>
| github.com | SOA  | 8.8.8.8    | <null>       | dns1.p08.nsone.net.      | <null>   | <null>    | <null>
| github.com | TXT  | 8.8.8.8    | <null>       | <null>                   | <null>   | <null>    | 66.78.69.170 ip4:166.78.71.131 ip4:167.89.101.2 ip4:167.89.101.192/28 ip4:192.254.112.60 ip4:192.254.112.98/31 i
| github.com | TXT  | 8.8.8.8    | <null>       | <null>                   | <null>   | <null>    | MS=6BF03E6AF5CB689E315FB6199603BABF2C88D805
| github.com | TXT  | 8.8.8.8    | <null>       | <null>                   | <null>   | <null>    | MS=ms44452932
| github.com | TXT  | 8.8.8.8    | <null>       | <null>                   | <null>   | <null>    | MS=ms58704441
| github.com | TXT  | 8.8.8.8    | <null>       | <null>                   | <null>   | <null>    | adobe-idp-site-verification=b92c9e999aef825edc36e0a3d847d2dbad5b2fc0e05c79ddd7a16139b48ecf4b
| github.com | TXT  | 8.8.8.8    | <null>       | <null>                   | <null>   | <null>    | apple-domain-verification=RyQhdzTl6Z6x8ZP4
| github.com | TXT  | 8.8.8.8    | <null>       | <null>                   | <null>   | <null>    | atlassian-domain-verification=jjgw98AKv2aeoYFxiL/VFaoyPkn3undEssTRuMg6C/3Fp/iqhkV4HVV7WjYlVeF8
| github.com | TXT  | 8.8.8.8    | <null>       | <null>                   | <null>   | <null>    | docusign=087098e3-3d46-47b7-9b4e-8a23028154cd
| github.com | TXT  | 8.8.8.8    | <null>       | <null>                   | <null>   | <null>    | facebook-domain-verification=39xu4jzl7roi7x0n93ldkxjiaarx50
| github.com | TXT  | 8.8.8.8    | <null>       | <null>                   | <null>   | <null>    | google-site-verification=UTM-3akMgubp6tQtgEuAkYNYLyYAvpTnnSrDMWoDR3o
| github.com | TXT  | 8.8.8.8    | <null>       | <null>                   | <null>   | <null>    | krisp-domain-verification=ZlyiK7XLhnaoUQb2hpak1PLY7dFkl1WE
| github.com | TXT  | 8.8.8.8    | <null>       | <null>                   | <null>   | <null>    | loom-site-verification=f3787154f1154b7880e720a511ea664d
| github.com | TXT  | 8.8.8.8    | <null>       | <null>                   | <null>   | <null>    | stripe-verification=f88ef17321660a01bab1660454192e014defa29ba7b8de9633c69d6b4912217f
| github.com | TXT  | 8.8.8.8    | <null>       | <null>                   | <null>   | <null>    | v=spf1 ip4:192.30.252.0/22 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.goog
+------------+------+------------+--------------+--------------------------+----------+-----------+-----------------------------------------------------------------------------------------------------------------

So I believe the net_dns_record table should fallback to EDNS and/or TCP, similar to https://github.com/miekg/exdns/blob/master/q/q.go#L385-L407.

I haven't had a chance to fully implement fallback yet, but I'm planning to look into this functionality hopefully sometime this week (and ensure that there are no security implications with its addition).

@massyn If you're interested, can you please test the add-dns-truncated-fallback branch locally and see if you can get the missing DNS records for github.com? If you get the records, then I believe this is the root cause for your original issue and fallback implementation should solve it.

Thanks!

massyn commented 1 year ago

Hi @cbruno10 - I applied this branch to my instance, and it is working as expected. Let me know the next steps (if you'll create the pull request, or if you'd like me to initiate it).

github-actions[bot] commented 1 year ago

'This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.'

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

cbruno10 commented 1 year ago

Not stale

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

cbruno10 commented 1 year ago

As mentioned in https://github.com/turbot/steampipe-plugin-net/pull/57#issuecomment-1660962853, this issue is on our overall TODO list but is not planned at the moment. There's a workaround in https://github.com/turbot/steampipe-plugin-net/commit/869fe3e5027047d50b7e74342c4cfd52365f7b72, which increases the buffer size to 4096, but we'd like to fallback to TCP if we receive a truncated response for a more long term solution.

I'm closing this issue due to lack of prioritization, but if anyone is interested at implementing fallback, we can re-open this issue.