search

spamhaus / rbldnsd

A small and fast DNS daemon especially made to serve DNSBL zones.
https://rbldnsd.io/
GNU General Public License v2.0
57 stars 12 forks source link

Test suite requires python-2.x #16

Closed orlitzky closed 3 years ago

orlitzky commented 4 years ago

Now that python-2.x is finally being EOLed , for real this time, we swear -- from all Linux distributions -- the rbldnsd test suite needs to be updated to work with python-3.x.

ammammita commented 3 years ago

Hello,

there is a branch called python3-tests If you'd be so kind to have a look, it should address this ticket.

orlitzky commented 3 years ago

I get a bunch of warnings with python-3.8 on Gentoo, but the test suite does ultimately pass:

Running tests.py
Can not deduce size of pointer. Assuming pointer size of 8.
/usr/lib/python3.8/unittest/case.py:633: ResourceWarning: unclosed file <_io.BufferedRandom name=5>
  method()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
.s.s/usr/lib/python3.8/unittest/case.py:633: ResourceWarning: unclosed file <_io.BufferedRandom name=7>
  method()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
......./usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=4 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=5 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=6 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=7 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=8 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=9 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=10 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=11 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=12 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=13 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=14 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=15 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=16 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=17 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=18 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/site-packages/DNS/Lib.py:155: ResourceWarning: unclosed file <_io.FileIO name=19 mode='rb+' closefd=True>
  for label in name.split('.'):
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/unittest/case.py:633: ResourceWarning: unclosed file <_io.BufferedRandom name=21>
  method()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
............./usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=3 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=4 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=5 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=6 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=7 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=8 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=9 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=10 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=11 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=12 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=13 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=14 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=15 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=16 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=17 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=18 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=19 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=20 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=21 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=22 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=23 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=24 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=25 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.8/tempfile.py:147: ResourceWarning: unclosed file <_io.FileIO name=26 mode='rb+' closefd=True>
  letters = [choose(c) for dummy in range(8)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
......
----------------------------------------------------------------------
Ran 30 tests in 6.439s

OK (skipped=2)
orlitzky commented 3 years ago

Oh, and I noticed you changed a warning from "The py3dns library is not installed..." to "The python3-dns library is not installed..." That misled me for a moment, because AFAIK the name "py3dns" is correct (i.e. that's what I needed to install). This is its project page: https://git.launchpad.net/py3dns

ammammita commented 3 years ago

i thought the same and i tested the patch with py3dns too.

Then, working on the travis pull request, i noticed it was importing only python-dns (not pydns) and it was working, so i gave a try to python3-dns and that seems a better idea other than py3dns that requires a separate installation.

Regarding the warnings: i noticed them too but i have not enough experience with python to fix them. A patch would be appreciated.

orlitzky commented 3 years ago

I think the warnings are just a matter of calling close() on the temporary files. This makes the errors go away, but is probably not correct since I didn't really pay attention to where/how those files are being used or cleaned up.

diff --git a/rbldnsd.py b/rbldnsd.py
index 79ef6d2..8134047 100644
--- a/rbldnsd.py
+++ b/rbldnsd.py
@@ -27,13 +27,16 @@ $NS 1d ns0.example.org

 class ZoneFile(object):
     def __init__(self, lines=None, no_header=False):
-        self._file = NamedTemporaryFile()
+        self._file = NamedTemporaryFile(delete=False)
         if not no_header:
             self._file.write(bytes(DUMMY_ZONE_HEADER, encoding = 'utf-8'))
         if lines is not None:
             self.writelines(lines)
         self._file.flush()

+    def __del__(self):
+        self._file.close()
+
     @property
     def name(self):
         return self._file.name
@@ -157,6 +160,8 @@ class Rbldnsd(object):
                     raise DaemonError("can not kill stop rbldnsd")
                 time.sleep(0.1)

+        self._stdout.close()
+
         self._daemon = None
         if daemon.returncode != 0:
             raise DaemonError("rbldnsd exited with code %d"
diff --git a/test_acl.py b/test_acl.py
index 84c9f55..f071c08 100644
--- a/test_acl.py
+++ b/test_acl.py
@@ -44,9 +44,10 @@ no_ipv6 = not _have_ipv6()
 def daemon(acl, addr='localhost'):
     """ Create an Rbldnsd instance with given ACL
     """
-    acl_zone = NamedTemporaryFile()
+    acl_zone = NamedTemporaryFile(delete=False)
     acl_zone.writelines(bytes("%s\n" % line, encoding='utf8') for line in acl)
     acl_zone.flush()
+    acl_zone.close()

     dnsd = Rbldnsd(daemon_addr=addr)
     dnsd.add_dataset('acl', acl_zone)
diff --git a/test_btrie.py b/test_btrie.py
index 51a61ed..6f6e450 100644
--- a/test_btrie.py
+++ b/test_btrie.py
@@ -32,7 +32,7 @@ def deduce_pointer_size(makefile='./Makefile'):
     cc = make_vars['CC']
     cflags = make_vars['CFLAGS']

-    test_c = NamedTemporaryFile(suffix=".c")
+    test_c = NamedTemporaryFile(suffix=".c", delete=False)
     test_c.write(r'''
 #include <stdio.h>
 #ifndef __SIZEOF_POINTER__
@@ -44,6 +44,7 @@ int main () {
 }
 ''')
     test_c.flush()
+    test_c.close()
     src = test_c.name

     try:
@@ -123,6 +124,9 @@ class CaptureOutput(object):
     def __init__(self):
         self._file = TemporaryFile()

+    def __del__(self):
+        self._file.close()
+
     def fileno(self):
         return self._file.fileno()
ammammita commented 3 years ago

Thank you Michael. I applied your patch to the python3-tests branch and now it runs perfectly!