Problems with AdvancedForwarding.App

[Potterli20]

Is there a problem with this configuration file? { "enableForwarding": true, "forwarders": [ { "name": "local-tcp", "dnssecValidation": true, "forwarderProtocol": "tcp", "forwarderAddresses": [ "127.0.0.1:56", "127.0.0.1:57", "127.0.0.1:58", "127.0.0.1:59", "127.0.0.1:60" ] }, { "name": "local-udp", "dnssecValidation": true, "forwarderProtocol": "udp", "forwarderAddresses": [ "127.0.0.1:56", "127.0.0.1:57", "127.0.0.1:58", "127.0.0.1:59", "127.0.0.1:60" ] } ], "networkGroupMap": { "0.0.0.0/0": "everyone", "[::]/0": "everyone" }, "groups": [ { "name": "everyone", "enableForwarding": true, "forwardings": [ { "forwarders": [ "local-tcp", "local-udp" ], "domains": [ "*" ] } ] } ] }

[ShreyasZare]

I tested the exact same config that you have on my local test setup and its working as expected without any issues.

It's so strange, I write local dns with a file, my configuration file does not take effect, even if the cache is cleaned, it does not work, and it takes effect only when I write forwarders

He has a request, but does not parse the IP when using dig

I would suggest that you run tcpdump -i any -w dump.pcap on the server with the same config and test it. After running the test query, stop tcpdump by pressing ctrl+c, and open the dump.pcap file in Wireshark to confirm if the DNS server is using your config. You can also share that file here or via email for analysis.

[Potterli20]

This is my other DNS server, all root DNS servers, no shunt files

[Potterli20]

Detecting dns diversion files is slow on podman containers. My dns file size is about 550m. https://github.com/Potterli20/file/releases/download/dns-hosts-all/dnshosts-all-adguardhome-blacklist_full_combine.txt

[Potterli20]

Also I would like to ask about support for parallel requests for doh, dot, doh3, doq?

[ShreyasZare]

Detecting dns diversion files is slow on podman containers. My dns file size is about 550m. https://github.com/Potterli20/file/releases/download/dns-hosts-all/dnshosts-all-adguardhome-blacklist_full_combine.txt

The file size of 550MB is quite large so its going to take a while for the parsing. The issue mostly with the slowness is due to time it takes for the file to be read from disk, the actual parsing does not take much CPU. Also, check how much RAM there is available since the file is loaded into memory and thus if there is low memory then it will cause swapping which will use disk and slow everything down.

[ShreyasZare]

Also I would like to ask about support for parallel requests for doh, dot, doh3, doq?

If you have more than one forwarder configured then the requests are done in parallel. But the Forwarder Concurrency option in Settings > Proxy & Forwarders section is used so you may edit that option to allow more parallel requests as needed.

[Potterli20]

Detecting dns diversion files is slow on podman containers. My dns file size is about 550m. https://github.com/Potterli20/file/releases/download/dns-hosts-all/dnshosts-all-adguardhome-blacklist_full_combine.txt

The file size of 550MB is quite large so its going to take a while for the parsing. The issue mostly with the slowness is due to time it takes for the file to be read from disk, the actual parsing does not take much CPU. Also, check how much RAM there is available since the file is loaded into memory and thus if there is low memory then it will cause swapping which will use disk and slow everything down.

I'm also slow with 220m files, I'm now using the container podman. it's also slow to read, I see that the memory and cpu is not full, it's all around 5%, except just opening the container goes to around 100%, normal parsing wouldn't go to 10%. So a bit puzzled as to how to optimize the read data.

[Potterli20]

Also I would like to ask about support for parallel requests for doh, dot, doh3, doq?

If you have more than one forwarder configured then the requests are done in parallel. But the Forwarder Concurrency option in Settings > Proxy & Forwarders section is used so you may edit that option to allow more parallel requests as needed.

Is that a parallel request on advanced forwarding? I've set >dns upstream there to adjust it to 10 as well, but it doesn't feel like it's working.

[Potterli20]

The # that follows in the dns triage file is not in effect. https://github.com/AdguardTeam/AdGuardHome/wiki/Configuration#upstreams-for-domains

[ShreyasZare]

I'm also slow with 220m files, I'm now using the container podman. it's also slow to read, I see that the memory and cpu is not full, it's all around 5%, except just opening the container goes to around 100%, normal parsing wouldn't go to 10%. So a bit puzzled as to how to optimize the read data.

If you have SSD then keep the file on it and try. The issue mostly is slow IO from HDD but it could be something else too.

[ShreyasZare]

Also I would like to ask about support for parallel requests for doh, dot, doh3, doq?

If you have more than one forwarder configured then the requests are done in parallel. But the Forwarder Concurrency option in Settings > Proxy & Forwarders section is used so you may edit that option to allow more parallel requests as needed.

Is that a parallel request on advanced forwarding? I've set >dns upstream there to adjust it to 10 as well, but it doesn't feel like it's working.

The parallel query feature is tested and it does send parallel requests. You can test that by running packet capture with a test setup to confirm.

[ShreyasZare]

The # that follows in the dns triage file is not in effect. https://github.com/AdguardTeam/AdGuardHome/wiki/Configuration#upstreams-for-domains

I tested the following sample config and it works as expected i.e. the # will cause www.example.com to get forwarded to 8.8.8.8 while example.com gets forwarded to 1.1.1.1.

8.8.8.8
[/example.com/]1.1.1.1
[/www.example.com/]#

[Potterli20]

The # that follows in the dns triage file is not in effect. https://github.com/AdguardTeam/AdGuardHome/wiki/Configuration#upstreams-for-domains

I tested the following sample config and it works as expected i.e. the # will cause www.example.com to get forwarded to 8.8.8.8 while example.com gets forwarded to 1.1.1.1.

8.8.8.8
[/example.com/]1.1.1.1
[/www.example.com/]#

Maybe the file is too big, it feels like it's not taking effect. I'll test it later.

[Potterli20]

I'm also slow with 220m files, I'm now using the container podman. it's also slow to read, I see that the memory and cpu is not full, it's all around 5%, except just opening the container goes to around 100%, normal parsing wouldn't go to 10%. So a bit puzzled as to how to optimize the read data.

If you have SSD then keep the file on it and try. The issue mostly is slow IO from HDD but it could be something else too.

I'll change the container path and see. My system disk is ssd.

[Potterli20]

This is the diary, already running on the ssd 2023-08-07.log

[ShreyasZare]

This is the diary, already running on the ssd 2023-08-07.log

Thanks for the logs.

System.FormatException: Invalid AdGuard Upstreams config file format: missing ']' bracket.

The error seen in the log suggest that there is a closing bracket missing in the adguard file. So, fixing this will make it work as expected.

The https://doh.dns.apple.com/dns-query ([2403:300:a26:3000::192]) DoH server does not seem to answer any requests. For any requests it returns REFUSED response.

TechnitiumLibrary.Net.Dns.DnsClientException: No IP address was found for name server: dns64.dns.google

For using the https://dns64.dns.google/dns-query DoH server, you should either specify its IPv6 address in brackets OR you must enable Prefer IPv6 option in the Settings so that the DNS server will resolve the IPv6 address by default to make it work.

[Potterli20]

This is the diary, already running on the ssd 2023-08-07.log

Thanks for the logs.

System.FormatException: Invalid AdGuard Upstreams config file format: missing ']' bracket.

The error seen in the log suggest that there is a closing bracket missing in the adguard file. So, fixing this will make it work as expected.

The https://doh.dns.apple.com/dns-query ([2403:300:a26:3000::192]) DoH server does not seem to answer any requests. For any requests it returns REFUSED response.

TechnitiumLibrary.Net.Dns.DnsClientException: No IP address was found for name server: dns64.dns.google

For using the https://dns64.dns.google/dns-query DoH server, you should either specify its IPv6 address in brackets OR you must enable Prefer IPv6 option in the Settings so that the DNS server will resolve the IPv6 address by default to make it work.

This file is working fine at adgh. The apple doh can only use apple domains. The podman container is also on ipv6 and the public network is also ipv6. Does ipv6 work with the dns64 app?

[ShreyasZare]

This file is working fine at adgh.

Maybe their parser is ignoring this syntax error and thus its working. I would recommend that you check the file syntax once and add the missing ']' bracket.

The apple doh can only use apple domains.

In that case, using apple's DoH server as a forwarder will generate these errors. Its recommended not to use it to avoid performance issues.

The podman container is also on ipv6 and the public network is also ipv6.

I was mentioning the specific error for resolving dns64.dns.google domain for the DoH address. If you enable Prefer IPv6 option in the DNS server's Settings > General section then this error will go away and the DoH URL will work as expected.

Does ipv6 work with the dns64 app?

The DNS64 app is useful only if you have a IPv6 only internet and your ISP has NAT64 service. Otherwise its of no use.

[Potterli20]

Maybe their parser is ignoring this syntax error and thus its working. I would recommend that you check the file syntax once and add the missing ']' bracket.

okok I'll see if there are any problems with the generation script. My triage file generation is scripted.

In that case, using apple's DoH server as a forwarder will generate these errors. Its recommended not to use it to avoid performance issues.

The apple doh service, I am writing a separate dns file, I only want to use the apple domain name alone to go to this file, in addition to the configuration file is a little too long over the top, I did not set up in the configuration file, just write on the dns file.

I was mentioning the specific error for resolving dns64.dns.google domain for the DoH address. If you enable Prefer IPv6 option in the DNS server's Settings > General section then this error will go away and the DoH URL will work as expected.

Yes, I have ipv6, so it's only right that there shouldn't be an error.

[Potterli20]

On my end, the test script is normal and formatted properly. I have a feeling it's not loading completely.

[ShreyasZare]

On my end, the test script is normal and formatted properly. I have a feeling it's not loading completely.

If possible please share one sample file which is failing to load so that I can test it locally. You can share here or over email.

[Potterli20]

On my end, the test script is normal and formatted properly. I have a feeling it's not loading completely.

If possible please share one sample file which is failing to load so that I can test it locally. You can share here or over email.

I'll e-mail it to you. It's a little big for me. 550m.

[ShreyasZare]

On my end, the test script is normal and formatted properly. I have a feeling it's not loading completely.

If possible please share one sample file which is failing to load so that I can test it locally. You can share here or over email.

I'll e-mail it to you. It's a little big for me. 550m.

Do zip the file when sending it.

[Potterli20]

On my end, the test script is normal and formatted properly. I have a feeling it's not loading completely.

If possible please share one sample file which is failing to load so that I can test it locally. You can share here or over email.

I'll e-mail it to you. It's a little big for me. 550m.

Do zip the file when sending it.

okl

[ShreyasZare]

On my end, the test script is normal and formatted properly. I have a feeling it's not loading completely.

If possible please share one sample file which is failing to load so that I can test it locally. You can share here or over email.

I'll e-mail it to you. It's a little big for me. 550m.

Thanks for the sample files. Both of them are loading without any errors on my laptop test setup. The 521MB file loads in 21 seconds on my laptop test setup which is a reasonable amount of time for a file of that size.

The only error that I got in DNS logs was TechnitiumLibrary.Net.Dns.DnsClientException: No IP address was found for name server: dns64.dns.google. This error too was fixed by enabling Prefer IPv6 option in DNS server's Settings and the domain names in the sample file were resolving as expected after that.

[Potterli20]

Then I don't quite understand, my cpu is j4125 system is debian dns is getting it in podman container, I load it in about 30 seconds, but when I write a script to update it, he doesn't load it. Even after clearing the dns program cache.

[ShreyasZare]

Then I don't quite understand, my cpu is j4125 system is debian dns is getting it in podman container, I load it in about 30 seconds, but when I write a script to update it, he doesn't load it. Even after clearing the dns program cache.

My test laptop is 7 year old with Intel i7 6700HQ and loading it in 21 seconds. If you update the file on disk, it will be automatically reloaded within 60 seconds. There is a timer which checks for file last modified date and if that has changed then reloads it. So, it may not load instantly and will need waiting a while for the timer to detect the file changes.

[Potterli20]

Then I don't quite understand, my cpu is j4125 system is debian dns is getting it in podman container, I load it in about 30 seconds, but when I write a script to update it, he doesn't load it. Even after clearing the dns program cache.

My test laptop is 7 year old with Intel i7 6700HQ and loading it in 21 seconds. If you update the file on disk, it will be automatically reloaded within 60 seconds. There is a timer which checks for file last modified date and if that has changed then reloads it. So, it may not load instantly and will need waiting a while for the timer to detect the file changes.

Well bla bla, I can't get it to work with a physical debian operating system, nor with a virtual machine, nor with a container. I don't know of any program that works.