search

glideinWMS / glideinwms

The glideinWMS Project
http://tinyurl.com/glideinwms
Apache License 2.0
16 stars 46 forks source link

Missing dependencies and unit tests failing on Alma8 w/ python39 #177

Closed mambelli closed 2 years ago

mambelli commented 2 years ago

Describe the bug Unit tests are failing on ALma8 with python3.9.

PYUNITTEST_FILES_CHECKED_COUNT=42
PYUNITTEST_ERROR_FILES="./test_lib_fork.py ./test_lib_x509Support.py ./test_OSG_autoconf.py ./test_factory_glideFactoryEntry.py ./test_frontend_element.py ./test_frontend_glideinFrontendElement.py ./test_glideinFrontendPlugins.py"
PYUNITTEST_ERROR_FILES_COUNT=7
PYUNITTEST_ERROR_COUNT=2

The installation of python39-cryptography fixed some tests that were failing with ModuleNotFoundError: No module named 'cryptography'. The RPM dependencies should probably target a specific python version.

PYUNITTEST_FILES_CHECKED_COUNT=42
PYUNITTEST_ERROR_FILES="./test_lib_fork.py ./test_lib_x509Support.py ./test_OSG_autoconf.py"
PYUNITTEST_ERROR_FILES_COUNT=3
PYUNITTEST_ERROR_COUNT=2

At times also test_lib_symCrypto.py seems to fail, triggered by the changes in branch issue_155_2 but not all the time, and showing errors due to encoding and seemingly not related to the changes in the branch. So I'm adding it here even if have not observed that problem in the master branch:

PYUNITTEST_FILES_CHECKED_COUNT=42
PYUNITTEST_ERROR_FILES="./test_lib_fork.py ./test_lib_x509Support.py ./test_OSG_autoconf.py ./test_lib_symCrypto.py"
PYUNITTEST_ERROR_FILES_COUNT=4
PYUNITTEST_ERROR_COUNT=3

To Reproduce Install OSG 3.5 and the vofrontend (and dependencies) on a Alma8 host (I used Fermicloud). Install also yum install python39-cryptography to avoid those errors. And run the unit tests

Expected behavior All tests should be successful

Screenshots output/gwms.LOCAL.pyunittest.d/gwms.LOCAL.pyunittest.d.test_lib_fork.txt

Running tests...
----------------------------------------------------------------------
..E...Traceback (most recent call last):
  File "/opt/gwms/glideinwms-mmascher/unittests/././test_lib_fork.py", line 242, in <module>
    unittest.main(testRunner=xmlrunner.XMLTestRunner(output="unittests-reports"))
  File "/usr/lib64/python3.9/unittest/main.py", line 101, in __init__
    self.runTests()
  File "/usr/lib64/python3.9/unittest/main.py", line 271, in runTests
    self.result = testRunner.run(self.test)
  File "/opt/gwms/venv-3.9.7/lib64/python3.9/site-packages/xmlrunner/xmlrunner.py", line 416, in run
    test(result)
  File "/usr/lib64/python3.9/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib64/python3.9/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib64/python3.9/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib64/python3.9/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib64/python3.9/unittest/case.py", line 651, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib64/python3.9/unittest/case.py", line 592, in run
    self._callTestMethod(testMethod)
  File "/usr/lib64/python3.9/unittest/case.py", line 550, in _callTestMethod
    method()
  File "/opt/gwms/glideinwms-mmascher/unittests/././test_lib_fork.py", line 183, in test_bounded_fork_and_collect_use_epoll
    results = self.fork_manager.bounded_fork_and_collect(max_forks=50, log_progress=True, sleep_time=0.1)
  File "/opt/gwms/glideinwms/lib/fork.py", line 369, in bounded_fork_and_collect
    post_work_info_subset = fetch_ready_fork_result_list(pipe_ids)
  File "/opt/gwms/glideinwms/lib/fork.py", line 239, in fetch_ready_fork_result_list
    readable_fds = [i[0] for i in poll_obj.poll(POLL_TIMEOUT)]
  File "/opt/gwms/glideinwms/lib/pidSupport.py", line 225, in termsignal
    raise KeyboardInterrupt("Received signal %s" % signr)
KeyboardInterrupt: Received signal 15

output/gwms.LOCAL.pyunittest.d/gwms.LOCAL.pyunittest.d.test_lib_x509Support.txt

Running tests...
----------------------------------------------------------------------
F
======================================================================
ERROR [0.057s]: test_extract_dn (__main__.TestExtractDN)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/gwms/glideinwms-mmascher/unittests/././test_lib_x509Support.py", line 37, in test_extract_dn
    self.assertEqual(expected, extract_DN(fname))
AssertionError: '= org, DC = opensciencegrid, O = Open Sci[49 chars].gov' != '/DC=org/DC=opensciencegrid/O=Open Science[39 chars].gov'
- = org, DC = opensciencegrid, O = Open Science Grid, OU = Services, CN = fermicloud308.fnal.gov
?  -   ^^  - -               ^^ - -                 ^^  - -        ^^  - -
+ /DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fermicloud308.fnal.gov
? +++    ^                  ^                   ^           ^

----------------------------------------------------------------------
Ran 1 test in 0.057s

FAILED (errors=1)

output/gwms.LOCAL.pyunittest.d/gwms.LOCAL.pyunittest.d.test_OSG_autoconf.txt

Running tests...
----------------------------------------------------------------------
.E
======================================================================
ERROR [0.000s]: test_get_information_internal (__main__.TestOSGAutoconf)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/gwms/glideinwms-mmascher/unittests/././test_OSG_autoconf.py", line 120, in test_get_information_internal
    self.assertEqual(get_information_internal(info), expected_out)
  File "/opt/gwms/glideinwms/factory/tools/OSG_autoconf.py", line 213, in get_information_internal
    result[resource][gatekeeper].setdefault(BEST_FIT_TAG, {})[entry] = get_bestfit_pilot(
  File "/opt/gwms/glideinwms/factory/tools/OSG_autoconf.py", line 97, in get_bestfit_pilot
    cpus = fractions.gcd(cpus, osg_catalog["CPUs"])
AttributeError: module 'fractions' has no attribute 'gcd'

----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (errors=1)

Generating XML reports...
WARNING! Site ANOTHER_SITE is in the whitelist file, but not in the collector. Retrieving it from old data (old OSG YAML or MISSING YAML), and saving it to the MISSING YAML
ERROR! Site ANOTHER_SITE is in the whitelist file, and I cant neither find it in the OSG YAML saved data, nor the MISSING YAML
WARNING! Site ANOTHER_SITE is in the whitelist file, but not in the collector. Retrieving it from old data (old OSG YAML or MISSING YAML), and saving it to the MISSING YAML
WARNING! CE ce01.sitename.edu of site SITE_NAME is in the whitelist file, but not in the collector. Retrieving it from old data (old OSG YAML or MISSING YAML), and saving it to the MISSING YAML
ERROR! CE ce01.sitename.edu of site SITE_NAME is in the whitelist file, and I cant neither find it in the OSG YAML saved data, nor the MISSING YAML
WARNING! CE ce01.sitename.edu of site SITE_NAME is in the whitelist file, but not in the collector. Retrieving it from old data (old OSG YAML or MISSING YAML), and saving it to the MISSING YAML
WARNING! CE ce01.sitename.edu of site SITE_NAME is in the whitelist file, but not in the collector. Retrieving it from old data (old OSG YAML or MISSING YAML), and saving it to the MISSING YAML
WARNING! CE ce02.sitename.edu of site SITE_NAME is in the whitelist file, but not in the collector. Retrieving it from old data (old OSG YAML or MISSING YAML), and saving it to the MISSING YAML
WARNING! CE ce01.sitename.edu of site SITE_NAME is in the whitelist file, but not in the collector. Retrieving it from old data (old OSG YAML or MISSING YAML), and saving it to the MISSING YAML
WARNING! CE ce02.sitename.edu of site SITE_NAME is in the whitelist file, but not in the collector. Retrieving it from old data (old OSG YAML or MISSING YAML), and saving it to the MISSING YAML

output/gwms.LOCAL.pyunittest.d/gwms.LOCAL.pyunittest.d.test_lib_symCrypto.txt

Running tests...
----------------------------------------------------------------------
.......S...E.E.E
======================================================================
ERROR [0.217s]: test_symmetric (__main__.TestSymAES256Key)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/gwms/glideinwms-mmascher/unittests/././test_lib_symCrypto.py", line 135, in test_symmetric
    def test_symmetric(self, data):
  File "/opt/gwms/venv-3.9.7/lib64/python3.9/site-packages/hypothesis/core.py", line 1246, in wrapped_test
    raise the_error_hypothesis_found
  File "/opt/gwms/glideinwms-mmascher/unittests/././test_lib_symCrypto.py", line 136, in test_symmetric
    data = defaults.force_bytes(data)
  File "/opt/gwms/glideinwms/lib/defaults.py", line 40, in force_bytes
    raise ValueError(
ValueError: Input was improperly converted into string (resulting in b'' characters added): b'

======================================================================
ERROR [0.059s]: test_symmetric (__main__.TestSymBlowfishKey)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/gwms/glideinwms-mmascher/unittests/././test_lib_symCrypto.py", line 161, in test_symmetric
    def test_symmetric(self, data):
  File "/opt/gwms/venv-3.9.7/lib64/python3.9/site-packages/hypothesis/core.py", line 1246, in wrapped_test
    raise the_error_hypothesis_found
  File "/opt/gwms/glideinwms-mmascher/unittests/././test_lib_symCrypto.py", line 162, in test_symmetric
    data = defaults.force_bytes(data)
  File "/opt/gwms/glideinwms/lib/defaults.py", line 40, in force_bytes
    raise ValueError(
ValueError: Input was improperly converted into string (resulting in b'' characters added): b'

======================================================================
ERROR [0.043s]: test_symmetric (__main__.TestSymDESKey)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/gwms/glideinwms-mmascher/unittests/././test_lib_symCrypto.py", line 214, in test_symmetric
    def test_symmetric(self, data):
  File "/opt/gwms/venv-3.9.7/lib64/python3.9/site-packages/hypothesis/core.py", line 1246, in wrapped_test
    raise the_error_hypothesis_found
  File "/opt/gwms/glideinwms-mmascher/unittests/././test_lib_symCrypto.py", line 215, in test_symmetric
    data = defaults.force_bytes(data)
  File "/opt/gwms/glideinwms/lib/defaults.py", line 40, in force_bytes
    raise ValueError(
ValueError: Input was improperly converted into string (resulting in b'' characters added): b'

----------------------------------------------------------------------
Ran 16 tests in 0.466s

FAILED (errors=3, skipped=1)

Generating XML reports...
Falsifying example: test_symmetric(
    data="b'", self=<__main__.TestSymAES256Key testMethod=test_symmetric>,
)
Falsifying example: test_symmetric(
    data="b'", self=<__main__.TestSymBlowfishKey testMethod=test_symmetric>,
)
Falsifying example: test_symmetric(
    data="b'", self=<__main__.TestSymDESKey testMethod=test_symmetric>,
)

Desktop (please complete the following information):

Additional context Add any other context or supporting files about the problem here.

mambelli commented 2 years ago

Only the lib/fork.py timeout is left to solve.

mambelli commented 2 years ago

After skipping the epoll test that was timing out, there is another fork error:

[rami_l@fermicloud490 work39]$ cat output/gwms.LOCAL.pyunittest.d/gwms.LOCAL.pyunittest.d.test_lib_fork.txt

Running tests...
----------------------------------------------------------------------
..E...S......
======================================================================
ERROR [10.008s]: test_fetch_ready_fork_result_list (__main__.TestFetchReadyForkResultList)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rami_l/work39/glideinwms/unittests/./test_lib_fork.py", line 116, in test_fetch_ready_fork_result_list
    result = fetch_ready_fork_result_list(pipe_ids)
  File "/home/rami_l/work39/glideinwms/lib/fork.py", line 267, in fetch_ready_fork_result_list
    poll_obj.unregister(fd)  # Is this needed? the poll object is no more used, next time will be a new one
OSError: [Errno 9] Bad file descriptor

----------------------------------------------------------------------
Ran 13 tests in 61.548s

FAILED (errors=1, skipped=1)

Generating XML reports...
j00961415 commented 2 years ago

Fork error after skipping epoll solved changed the key for range to i

j00961415 commented 2 years ago

PYUNITTEST_FILES_CHECKED_COUNT=84 PYUNITTEST_ERROR_FILES="" PYUNITTEST_ERROR_FILES_COUNT=0 PYUNITTEST_ERROR_COUNT=0 BRANCH=LOCAL PYUNITTEST_TIME=548 PYUNITTEST=success

[rami_l@fermicloud490 work39]$

j00961415 commented 2 years ago

for key in range(1, 5): pipe_ids[key] = fork_in_bg(sleep_fn, svl) expected[key] = str(svl) for i in range(1, 5): pipe_ids[i] = fork_in_bg(sleep_fn, svl) expected[i] = str(svl)

    result = fetch_fork_result_list(pipe_ids)
    self.assertTrue(expected, result)

@@ -109,11 +109,11 @@ def test_fetch_ready_fork_result_list(self): pipe_ids = {} expected = {} svl = 10 for key in range(1, 5): pipe_ids[key] = fork_in_bg(sleep_fn, svl) expected[key] = str(svl) for i in range(1, 5): pipe_ids[i] = fork_in_bg(sleep_fn, svl) expected[i] = str(svl)

    **result = fetch_ready_fork_result_list(pipe_ids)**
j00961415 commented 2 years ago

reviewing source: https://docs.python.org/3/library/select.html to fix skipped epoll test

mambelli commented 2 years ago

The epoll problem was fixed in #193