External DNS environment test anomolies

[kitterma]

Describe the bug I would expect tests that are dependent on external environment factors to either pass or be skipped when they cannot be executed. I'm currently working on packaging dnspython 2.7.0 release candidate for Debian and am having problems with a few tests. The unusual part is that I'm traveling right now, so I'm limited to either hotel WiFi or tethering to my phone.

If I am connected to the hotel WiFi, the ddr tests fail:

    @pytest.mark.skipif(
        not tests.util.is_internet_reachable(), reason="Internet not reachable"
    )
    def test_basic_ddr_sync():
        for nameserver in ["1.1.1.1", "8.8.8.8"]:
            res = dns.resolver.Resolver(configure=False)
            res.nameservers = [nameserver]
            res.try_ddr()
            for nameserver in res.nameservers:
>               assert isinstance(nameserver, dns.nameserver.Nameserver)
E               AssertionError: assert False
E                +  where False = isinstance('1.1.1.1', <class 'dns.nameserver.Nameserver'>)
E                +    where <class 'dns.nameserver.Nameserver'> = <module 'dns.nameserver' from '/usr/lib/python3/dist-packages/dns/nameserver.py'>.Nameserver
E                +      where <module 'dns.nameserver' from '/usr/lib/python3/dist-packages/dns/nameserver.py'> = dns.nameserver

tests/test_ddr.py:25: AssertionError
_____________________________ test_basic_ddr_async _____________________________

    @pytest.mark.skipif(
        not tests.util.is_internet_reachable(), reason="Internet not reachable"
    )
    def test_basic_ddr_async():
        async def run():
            dns.asyncbackend._default_backend = None
            for nameserver in ["1.1.1.1", "8.8.8.8"]:
                res = dns.asyncresolver.Resolver(configure=False)
                res.nameservers = [nameserver]
                await res.try_ddr()
                for nameserver in res.nameservers:
                    assert isinstance(nameserver, dns.nameserver.Nameserver)
                    assert nameserver.kind() != "Do53"

>       asyncio.run(run())

tests/test_ddr.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

If am tethered via my phone the dangling target tests fail:

_____________________ AsyncTests.testCanonicalNameDangling _____________________

self = <tests.test_async.AsyncTests testMethod=testCanonicalNameDangling>

    @unittest.skipIf(
        _systemd_resolved_present or _is_docker, "systemd-resolved or docker in use"
    )
    def testCanonicalNameDangling(self):
        name = dns.name.from_text("dangling-cname.dnspython.org")
        cname = dns.name.from_text("dangling-target.dnspython.org")

        async def run():
            return await dns.asyncresolver.canonical_name(name)

>       self.assertEqual(self.async_run(run), cname)
E       AssertionError: <DNS name dangling-cname.dnspython.org.> != <DNS name dangling-target.dnspython.org.>

tests/test_async.py:280: AssertionError
___________________ TrioAsyncTests.testCanonicalNameDangling ___________________

self = <tests.test_async.TrioAsyncTests testMethod=testCanonicalNameDangling>

    @unittest.skipIf(
        _systemd_resolved_present or _is_docker, "systemd-resolved or docker in use"
    )
    def testCanonicalNameDangling(self):
        name = dns.name.from_text("dangling-cname.dnspython.org")
        cname = dns.name.from_text("dangling-target.dnspython.org")

        async def run():
            return await dns.asyncresolver.canonical_name(name)

>       self.assertEqual(self.async_run(run), cname)
E       AssertionError: <DNS name dangling-cname.dnspython.org.> != <DNS name dangling-target.dnspython.org.>

tests/test_async.py:280: AssertionError
_________________ LiveResolverTests.testCanonicalNameDangling __________________

self = <tests.test_resolver.LiveResolverTests testMethod=testCanonicalNameDangling>

    @unittest.skipIf(
        _systemd_resolved_present or _is_docker, "systemd-resolved or docker in use"
    )
    def testCanonicalNameDangling(self):
        name = dns.name.from_text("dangling-cname.dnspython.org")
        cname = dns.name.from_text("dangling-target.dnspython.org")
>       self.assertEqual(dns.resolver.canonical_name(name), cname)
E       AssertionError: <DNS name dangling-cname.dnspython.org.> != <DNS name dangling-target.dnspython.org.>

tests/test_resolver.py:808: AssertionError
=========================== short test summary info ============================
FAILED tests/test_async.py::AsyncTests::testCanonicalNameDangling - Assertion...
FAILED tests/test_async.py::TrioAsyncTests::testCanonicalNameDangling - Asser...
FAILED tests/test_resolver.py::LiveResolverTests::testCanonicalNameDangling
======================= 3 failed, 1357 passed in 41.57s ====================

Since all tests pass in one environment or the other, I'm confident there's not an actual dnspython bug, but it would be nice to have the skip conditions work so that these are just skipped when they can't be run. Help!

To Reproduce Not sure. Looking for guidance.

Context (please complete the following information):

• dnspython version 2.6.1 and 2.7.0 release candidate
• Python version 3.12
• OS: Debian (unstable)

[rthalley]

I can delay the release a week no problem, no need to work on stuff on your vacation! I agree it's suboptimal that there are live tests at all, and that they can be flaky in some environments. Hotel networks are notoriously bad for DNS testing!

[rthalley]

See your other ticket for info about DDR. The dangling CNAME test failing is usually caused by some intermediate DNS software mangling the answer or not doing the usual thing. Dnspython tries to disable the test when it knows the system is broken, but I'm sure our list of broken things is not comprehensive. This test should also be made "non-live" in the future to solve this problem.